[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:22 dist-test-slave-r8r3 krb5kdc[27843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:22 dist-test-slave-r8r3 krb5kdc[27843](info): set up 2 sockets
Sep 25 04:33:22 dist-test-slave-r8r3 krb5kdc[27843](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 04:33:23.164399 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:23 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:23.299397 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44447
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:23.450521 27856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:23.450822 27856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:23.450899 27856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:23.455603 27856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:23.455720 27856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:23.455765 27856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:23.455804 27856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:23.455839 27856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:23.463635 27856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:23.464917 27856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:23.466265 27856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:23.513005 27862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:23.513113 27865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:23.513758 27856 server_base.cc:1034] running on GCE node
W20260925 04:33:23.515139 27863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:23.515648 27856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:23.516870 27856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:23.518043 27856 hybrid_clock.cc:648] HybridClock initialized: now 1790310803518025 us; error 30 us; skew 500 ppm
Sep 25 04:33:23 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:23.520764 27856 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:23.521888 27856 webserver.cc:460] Webserver started at http://127.27.46.254:36815/ using document root <none> and password file <none>
I20260925 04:33:23.522495 27856 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:23.522610 27856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:23.522878 27856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:23.525055 27856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "888cdac0157c4ae3b5cce9cd82c91dd7"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "3a5f89e5e36784b5e126c03ce12d545f"
server_key_iv: "ea4e50014de48d0d7105b210d0668691"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.525805 27856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "888cdac0157c4ae3b5cce9cd82c91dd7"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "3a5f89e5e36784b5e126c03ce12d545f"
server_key_iv: "ea4e50014de48d0d7105b210d0668691"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.529834 27856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:33:23.533807 27872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:23.535055 27856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:33:23.535271 27856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "888cdac0157c4ae3b5cce9cd82c91dd7"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "3a5f89e5e36784b5e126c03ce12d545f"
server_key_iv: "ea4e50014de48d0d7105b210d0668691"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.535454 27856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:23.559367 27856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:23.562489 27856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:23.562693 27856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:23.580677 27926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:44447 every 8 connection(s)
I20260925 04:33:23.580717 27856 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:44447
I20260925 04:33:23.581913 27856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:23.586434 27927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:23.588798 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 27856
I20260925 04:33:23.588968 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:23.590687 27835 external_mini_cluster.cc:1428] Setting key 1075a3cfc94dae9fcb0cea16cb077e75
I20260925 04:33:23.596647 27927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: Bootstrap starting.
I20260925 04:33:23.601193 27927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:23.602064 27927 log.cc:824] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:23.604641 27927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: No bootstrap required, opened a new log
Sep 25 04:33:23 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:23.612900 27927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } }
I20260925 04:33:23.613157 27927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:23.613253 27927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888cdac0157c4ae3b5cce9cd82c91dd7, State: Initialized, Role: FOLLOWER
I20260925 04:33:23.613848 27927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } }
I20260925 04:33:23.614022 27927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:23.614112 27927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:23.614310 27927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:23.615235 27927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } }
I20260925 04:33:23.615620 27927 leader_election.cc:302] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 888cdac0157c4ae3b5cce9cd82c91dd7; no voters:
I20260925 04:33:23.615983 27927 leader_election.cc:288] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:23.616046 27932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:23.616324 27932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 888cdac0157c4ae3b5cce9cd82c91dd7, State: Running, Role: LEADER
I20260925 04:33:23.616776 27932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } }
I20260925 04:33:23.617483 27927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:23.619464 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:23.595502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.27.46.254:44447)
0925 04:33:23.596072 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:23.596129 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:23.597315 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 04:33:23.601771 (+ 4456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:23.601787 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:23.602143 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:23.602442 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:23.605302 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.606362 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.608041 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.608583 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.608696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:23.612894 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:23.612905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:23.612917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:23.612943 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:23.614990 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:23.615559 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:23.615565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:23.615577 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:23.615685 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:23.616142 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:23.616146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:23.616147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:23.616693 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:23.616887 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:23.617647 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 04:33:23.617925 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":165,"threads_started":1}
I20260925 04:33:23.625644 27933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } } }
I20260925 04:33:23.625694 27934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 888cdac0157c4ae3b5cce9cd82c91dd7. Latest consensus state: current_term: 1 leader_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "888cdac0157c4ae3b5cce9cd82c91dd7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44447 } } }
I20260925 04:33:23.625831 27934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:23.625831 27933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7 [sys.catalog]: This master's current role is: LEADER
W20260925 04:33:23.625909 27946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:33:23.625979 27946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:33:23.626264 27949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:23.627198 27949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:23.633575 27949 catalog_manager.cc:1347] Generated new cluster ID: 63352b2cd7734c0397cd7b57969486df
I20260925 04:33:23.633638 27949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:23.642189 27949 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:23.643009 27949 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:23.651615 27949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 888cdac0157c4ae3b5cce9cd82c91dd7: Generated new TSK 0
I20260925 04:33:23.652309 27949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:23.688591 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:23.816843 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:23.817112 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:23.817191 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:23.821931 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:23.822023 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:23.822132 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:23.827770 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:23.828853 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:23.830024 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:23.839967 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:23.840014 27962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:23.840204 27953 server_base.cc:1034] running on GCE node
W20260925 04:33:23.840399 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:23.840811 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:23.841542 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:23.842706 27953 hybrid_clock.cc:648] HybridClock initialized: now 1790310803842698 us; error 31 us; skew 500 ppm
Sep 25 04:33:23 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:23.845988 27953 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:23.846895 27953 webserver.cc:460] Webserver started at http://127.27.46.193:44017/ using document root <none> and password file <none>
I20260925 04:33:23.847513 27953 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:23.847579 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:23.847815 27953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:23.850035 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "8d9daeec0a90415e930107896a68d87b"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "49e965dbf1703b1e6f719fc6616bca23"
server_key_iv: "6492d518f2001a1d7e65691f9b4d8168"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.850641 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "8d9daeec0a90415e930107896a68d87b"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "49e965dbf1703b1e6f719fc6616bca23"
server_key_iv: "6492d518f2001a1d7e65691f9b4d8168"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.854645 27953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:33:23.857290 27969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:23.859005 27953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:33:23.859125 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "8d9daeec0a90415e930107896a68d87b"
format_stamp: "Formatted at 2026-09-25 04:33:23 on dist-test-slave-r8r3"
server_key: "49e965dbf1703b1e6f719fc6616bca23"
server_key_iv: "6492d518f2001a1d7e65691f9b4d8168"
server_key_version: "encryptionkey@0"
I20260925 04:33:23.859256 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:23.878247 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:23.881347 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:23.881598 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:23.882413 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:23.883392 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:23.883445 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:23.883535 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:23.883574 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:23.898586 27953 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:38923
I20260925 04:33:23.898609 28086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:38923 every 8 connection(s)
I20260925 04:33:23.899606 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:23.905349 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 27953
I20260925 04:33:23.905429 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:23.905706 27835 external_mini_cluster.cc:1428] Setting key 63c34ff1db5a1134455bb5ec4b41e009
Sep 25 04:33:23 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:23.915804 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:23.901469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60713 (local address 127.27.46.254:44447)
0925 04:33:23.901664 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:23.901668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:23.902359 (+ 691us) server_negotiation.cc:407] Connection header received
0925 04:33:23.905008 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:23.905012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:23.905069 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:23.905182 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:23.907068 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.907757 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.909134 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.909613 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.909740 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:23.912260 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:23.912267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:23.912272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:23.912304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:23.913827 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:23.914317 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:23.914321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:23.914323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:23.914399 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:23.914742 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:23.914747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:23.914749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:23.914920 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:23.915025 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:23.915506 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 04:33:23.915633 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 04:33:23.916586 28089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:23.901763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44447 (local address 127.27.46.193:60713)
0925 04:33:23.902218 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:23.902259 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:23.904781 (+ 2522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:23.905359 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:23.905366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:23.906029 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:23.906741 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:23.906761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.907885 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.907888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:23.908968 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:23.908978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:23.909768 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:23.909771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:23.909911 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:23.910531 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:23.910571 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:23.912096 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:23.913943 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:23.913949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:23.913962 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:23.914215 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:23.914489 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:23.914493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:23.914495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:23.914628 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:23.915034 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:23.915043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:23.915273 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 04:33:23.915467 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 04:33:23.915774 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:33:23.918032 28087 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44447
I20260925 04:33:23.918255 28087 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:23.918713 28087 heartbeater.cc:511] Master 127.27.46.254:44447 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:23.920552 27890 ts_manager.cc:194] Registered new tserver with Master: 8d9daeec0a90415e930107896a68d87b (127.27.46.193:38923)
I20260925 04:33:23.922461 27890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:60713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:23.939179 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:24.074952 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:24.075181 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:24.075240 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:24.079100 28092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:24.079151 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:24.079206 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:24.084165 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:24.085127 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:24.086138 28092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:24.095691 28099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:24.095924 28098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:24.096066 28092 server_base.cc:1034] running on GCE node
W20260925 04:33:24.096159 28101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:24.096582 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:24.097298 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:24.098483 28092 hybrid_clock.cc:648] HybridClock initialized: now 1790310804098430 us; error 72 us; skew 500 ppm
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:24.101128 28092 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:24.101915 28092 webserver.cc:460] Webserver started at http://127.27.46.194:38951/ using document root <none> and password file <none>
I20260925 04:33:24.102450 28092 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:24.102504 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:24.102720 28092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:24.104718 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "ae70e1022e6643e2a12efdd1910e3f8e"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "422e033ec5ba8f370d79a975037797f7"
server_key_iv: "2e24cc966f2f1fca304c759e6fd00259"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.105320 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "ae70e1022e6643e2a12efdd1910e3f8e"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "422e033ec5ba8f370d79a975037797f7"
server_key_iv: "2e24cc966f2f1fca304c759e6fd00259"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.109220 28092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:33:24.111615 28108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.112867 28092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:24.112958 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "ae70e1022e6643e2a12efdd1910e3f8e"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "422e033ec5ba8f370d79a975037797f7"
server_key_iv: "2e24cc966f2f1fca304c759e6fd00259"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.113050 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:24.129439 28092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:24.132166 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:24.132344 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:24.132846 28092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:24.133766 28092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:24.133813 28092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.133855 28092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:24.133878 28092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.145589 28092 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:34641
I20260925 04:33:24.145630 28223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:34641 every 8 connection(s)
I20260925 04:33:24.146417 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:24.156105 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28092
I20260925 04:33:24.156224 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:24.156514 27835 external_mini_cluster.cc:1428] Setting key 68042914ef90a51d2753835f295dbddd
I20260925 04:33:24.159933 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.148198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:42695 (local address 127.27.46.254:44447)
0925 04:33:24.148332 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:24.148336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:24.149142 (+ 806us) server_negotiation.cc:407] Connection header received
0925 04:33:24.150429 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:24.150432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:24.150490 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:24.150594 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:24.151846 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.152387 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.153626 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.153846 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.153932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.156212 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:24.156217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:24.156221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:24.156250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:24.157912 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.158426 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.158433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.158437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.158523 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.158827 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.158832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.158837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.159079 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:24.159178 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:24.159618 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 04:33:24.159735 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:33:24.160667 28226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.148569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44447 (local address 127.27.46.194:42695)
0925 04:33:24.148993 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:24.149033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:24.150207 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:24.150785 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:24.150792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:24.151198 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:24.151695 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:24.151708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.152524 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.152528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:24.153454 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:24.153464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.153979 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.153981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:24.154115 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.154696 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:24.154712 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:24.156045 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:24.158038 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:24.158053 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:24.158066 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:24.158322 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:24.158612 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:24.158614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:24.158615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:24.158736 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:24.159176 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:24.159180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:24.159376 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 04:33:24.159561 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:33:24.159875 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260925 04:33:24.161870 28224 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44447
I20260925 04:33:24.162129 28224 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:24.162691 28224 heartbeater.cc:511] Master 127.27.46.254:44447 requested a full tablet report, sending...
I20260925 04:33:24.163826 27890 ts_manager.cc:194] Registered new tserver with Master: ae70e1022e6643e2a12efdd1910e3f8e (127.27.46.194:34641)
I20260925 04:33:24.164420 27890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:42695
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:24.185022 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:24.305938 28229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:24.306202 28229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:24.306280 28229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:24.310380 28229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:24.310484 28229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:24.310580 28229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:24.315687 28229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:24.316618 28229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:24.317739 28229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:24.327689 28237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:24.327725 28235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:24.327749 28234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:24.327831 28229 server_base.cc:1034] running on GCE node
I20260925 04:33:24.328274 28229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:24.328888 28229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:24.330070 28229 hybrid_clock.cc:648] HybridClock initialized: now 1790310804330040 us; error 40 us; skew 500 ppm
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:24.332790 28229 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:24.333695 28229 webserver.cc:460] Webserver started at http://127.27.46.195:40529/ using document root <none> and password file <none>
I20260925 04:33:24.334211 28229 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:24.334296 28229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:24.334520 28229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:24.336428 28229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "256a459b66cb4e9181d5a785bdebda07"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "d61ac3d9a964868112ad518d0ea9f80d"
server_key_iv: "5011be253737a4d966872f54d62e5637"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.336989 28229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "256a459b66cb4e9181d5a785bdebda07"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "d61ac3d9a964868112ad518d0ea9f80d"
server_key_iv: "5011be253737a4d966872f54d62e5637"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.340633 28229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:24.343073 28244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.344223 28229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:24.344374 28229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "256a459b66cb4e9181d5a785bdebda07"
format_stamp: "Formatted at 2026-09-25 04:33:24 on dist-test-slave-r8r3"
server_key: "d61ac3d9a964868112ad518d0ea9f80d"
server_key_iv: "5011be253737a4d966872f54d62e5637"
server_key_version: "encryptionkey@0"
I20260925 04:33:24.344499 28229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:24.364262 28229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:24.366966 28229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:24.367165 28229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:24.367731 28229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:24.368630 28229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:24.368701 28229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.368772 28229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:24.368813 28229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:24.381338 28229 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:44851
I20260925 04:33:24.381361 28362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:44851 every 8 connection(s)
I20260925 04:33:24.382261 28229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:24.390666 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28229
I20260925 04:33:24.390789 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:24.391062 27835 external_mini_cluster.cc:1428] Setting key fc30e9f3834eacab38877ba72483d227
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:24.395346 27929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.384060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:33881 (local address 127.27.46.254:44447)
0925 04:33:24.384203 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:24.384207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:24.385478 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 04:33:24.386387 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:24.386390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:24.386436 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:24.386512 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:24.387595 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.388119 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.389140 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.389334 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.389402 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.391876 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:24.391879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:24.391881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:24.391902 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:24.393371 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.393896 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.393900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.393902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.393979 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.394315 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.394318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.394320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.394518 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:24.394627 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:24.395074 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 04:33:24.395179 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:33:24.396076 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.384421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44447 (local address 127.27.46.195:33881)
0925 04:33:24.385326 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:24.385366 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:24.386198 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:24.386671 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:24.386678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:24.387011 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:24.387470 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:24.387482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.388219 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.388222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:24.389040 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:24.389048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.389450 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.389453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:24.389638 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.390226 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:24.390241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:24.391693 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:24.393528 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:24.393537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:24.393552 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:24.393792 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:24.394082 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:24.394085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:24.394086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:24.394215 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:24.394626 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:24.394632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:24.394852 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 04:33:24.395020 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:33:24.395317 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":579,"threads_started":1}
I20260925 04:33:24.397163 28363 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44447
I20260925 04:33:24.397387 28363 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:24.397846 28363 heartbeater.cc:511] Master 127.27.46.254:44447 requested a full tablet report, sending...
I20260925 04:33:24.398849 27890 ts_manager.cc:194] Registered new tserver with Master: 256a459b66cb4e9181d5a785bdebda07 (127.27.46.195:44851)
I20260925 04:33:24.399402 27890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:33881
I20260925 04:33:24.405418 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:24.430855 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.421843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.27.46.193:38923)
0925 04:33:24.422104 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:24.422111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:24.422136 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:33:24.422286 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:24.422299 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:24.422473 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:24.422617 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:24.423264 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.424065 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.424884 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.425254 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.425365 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.427626 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:24.427640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:24.427652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:24.427674 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:24.429047 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.429482 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.429486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.429487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.429625 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.429908 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.429913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.429914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.430149 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:24.430351 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:24.430563 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 04:33:24.430712 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
W20260925 04:33:24.431838 27998 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37374
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[27843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:24.457486 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:24.449409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.27.46.193:38923)
0925 04:33:24.449672 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:24.449675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:24.449819 (+ 144us) server_negotiation.cc:407] Connection header received
0925 04:33:24.449930 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:24.449933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:24.449976 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:24.450057 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:24.450714 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.451205 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.451979 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:24.452193 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:24.452268 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:24.454691 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:24.454694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:24.454696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:24.454718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:24.456111 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.456468 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.456470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.456471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.456539 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:24.456775 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:24.456777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:24.456778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:24.456925 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:24.457025 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:24.457275 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 04:33:24.457366 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260925 04:33:24.458559 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 27953
I20260925 04:33:24.464792 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28092
I20260925 04:33:24.471596 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28229
I20260925 04:33:24.477849 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 27856
2026-09-25T04:33:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2394 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[28377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[28377](info): set up 2 sockets
Sep 25 04:33:24 dist-test-slave-r8r3 krb5kdc[28377](info): commencing operation
krb5kdc: starting...
W20260925 04:33:25.517731 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:25 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:25.634148 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33655
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:33655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:25.741161 28390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:25.741390 28390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:25.741426 28390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:25.745280 28390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:25.745330 28390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:25.745349 28390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:25.745364 28390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:25.745378 28390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:25.750437 28390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:33655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:25.751322 28390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:25.752199 28390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:25.760180 28396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:25.760217 28397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:25.760222 28399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:25.760274 28390 server_base.cc:1034] running on GCE node
I20260925 04:33:25.760861 28390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:25.761945 28390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:25.763127 28390 hybrid_clock.cc:648] HybridClock initialized: now 1790310805763101 us; error 36 us; skew 500 ppm
Sep 25 04:33:25 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310805, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:25.765798 28390 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:25.766719 28390 webserver.cc:460] Webserver started at http://127.27.46.254:40943/ using document root <none> and password file <none>
I20260925 04:33:25.767270 28390 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:25.767364 28390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:25.767599 28390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:25.769630 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "acd439724339441d8ff1f61f25463218"
format_stamp: "Formatted at 2026-09-25 04:33:25 on dist-test-slave-r8r3"
server_key: "7de441b1f01849dc7cc5194253153646"
server_key_iv: "77b71f88338ae2579744116ef49d18ed"
server_key_version: "encryptionkey@0"
I20260925 04:33:25.770190 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "acd439724339441d8ff1f61f25463218"
format_stamp: "Formatted at 2026-09-25 04:33:25 on dist-test-slave-r8r3"
server_key: "7de441b1f01849dc7cc5194253153646"
server_key_iv: "77b71f88338ae2579744116ef49d18ed"
server_key_version: "encryptionkey@0"
I20260925 04:33:25.773789 28390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:25.776042 28406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:25.777191 28390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:25.777328 28390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "acd439724339441d8ff1f61f25463218"
format_stamp: "Formatted at 2026-09-25 04:33:25 on dist-test-slave-r8r3"
server_key: "7de441b1f01849dc7cc5194253153646"
server_key_iv: "77b71f88338ae2579744116ef49d18ed"
server_key_version: "encryptionkey@0"
I20260925 04:33:25.777549 28390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:25.787585 28390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:25.790144 28390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:25.790329 28390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:25.797206 28390 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:33655
I20260925 04:33:25.797232 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:33655 every 8 connection(s)
I20260925 04:33:25.798151 28390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:25.799657 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28390
I20260925 04:33:25.799769 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:25.800083 27835 external_mini_cluster.cc:1428] Setting key 57ce6b9bda3263f656ef3368793f1c6c
I20260925 04:33:25.801276 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:25.807308 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218: Bootstrap starting.
Sep 25 04:33:25 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:25.810003 28460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:25.810914 28460 log.cc:824] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:25.813094 28460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218: No bootstrap required, opened a new log
I20260925 04:33:25.814953 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:25.801511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.27.46.254:33655)
0925 04:33:25.801980 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:25.801995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:25.802032 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:33:25.803102 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:25.803127 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:25.803468 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:25.803843 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:25.804687 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:25.805756 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:25.806632 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:25.807126 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:25.807301 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:25.810010 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:25.810015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:25.810029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:25.810084 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:25.812011 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:25.812482 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:25.812487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:25.812492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:25.812602 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:25.812895 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:25.812899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:25.812901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:25.813368 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:25.813595 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:25.813910 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 04:33:25.814217 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260925 04:33:25.818508 28460 raft_consensus.cc:348] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } }
I20260925 04:33:25.818800 28460 raft_consensus.cc:374] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:25.818862 28460 raft_consensus.cc:729] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd439724339441d8ff1f61f25463218, State: Initialized, Role: FOLLOWER
I20260925 04:33:25.819319 28460 consensus_queue.cc:260] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } }
I20260925 04:33:25.819466 28460 raft_consensus.cc:388] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:25.819536 28460 raft_consensus.cc:482] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:25.819653 28460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:25.820885 28460 raft_consensus.cc:504] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } }
I20260925 04:33:25.821239 28460 leader_election.cc:302] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acd439724339441d8ff1f61f25463218; no voters:
I20260925 04:33:25.821610 28460 leader_election.cc:288] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:25.821765 28466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:25.822063 28466 raft_consensus.cc:686] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [term 1 LEADER]: Becoming Leader. State: Replica: acd439724339441d8ff1f61f25463218, State: Running, Role: LEADER
I20260925 04:33:25.822381 28466 consensus_queue.cc:237] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } }
I20260925 04:33:25.822897 28460 sys_catalog.cc:564] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:25.824108 28467 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acd439724339441d8ff1f61f25463218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } } }
I20260925 04:33:25.824249 28467 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:25.824544 28468 sys_catalog.cc:455] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd439724339441d8ff1f61f25463218. Latest consensus state: current_term: 1 leader_uuid: "acd439724339441d8ff1f61f25463218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acd439724339441d8ff1f61f25463218" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33655 } } }
I20260925 04:33:25.824636 28468 sys_catalog.cc:458] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:25.825064 28475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:25.828222 28475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:25.834059 28475 catalog_manager.cc:1347] Generated new cluster ID: 1a6f697e97524ad596aa63b40e624986
I20260925 04:33:25.834126 28475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:25.853863 28475 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:25.854774 28475 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:25.861532 28475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acd439724339441d8ff1f61f25463218: Generated new TSK 0
I20260925 04:33:25.862190 28475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:25.892239 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:26.000852 28487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:26.001096 28487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:26.001161 28487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:26.004974 28487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:26.005059 28487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:26.005151 28487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:26.010219 28487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:26.011219 28487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:26.012177 28487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:26.021247 28497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.021257 28494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.021299 28495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:26.021556 28487 server_base.cc:1034] running on GCE node
I20260925 04:33:26.021929 28487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:26.022523 28487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:26.023752 28487 hybrid_clock.cc:648] HybridClock initialized: now 1790310806023740 us; error 28 us; skew 500 ppm
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:26.026710 28487 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:26.027668 28487 webserver.cc:460] Webserver started at http://127.27.46.193:42701/ using document root <none> and password file <none>
I20260925 04:33:26.028223 28487 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:26.028285 28487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:26.028518 28487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:26.030558 28487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "f279f9f49ca1c3f9f191a51d9d8236f9"
server_key_iv: "7b43bd0237234b9baa74ec2f20bcc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.031116 28487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "f279f9f49ca1c3f9f191a51d9d8236f9"
server_key_iv: "7b43bd0237234b9baa74ec2f20bcc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.034823 28487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:26.037149 28504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.038408 28487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:26.038544 28487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "f279f9f49ca1c3f9f191a51d9d8236f9"
server_key_iv: "7b43bd0237234b9baa74ec2f20bcc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.038648 28487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:26.053645 28487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:26.056257 28487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:26.056455 28487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:26.057052 28487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:26.057956 28487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:26.058022 28487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.058080 28487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:26.058103 28487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.066329 28487 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44871
I20260925 04:33:26.066382 28621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44871 every 8 connection(s)
I20260925 04:33:26.067265 28487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:26.068298 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28487
I20260925 04:33:26.068377 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:26.068682 27835 external_mini_cluster.cc:1428] Setting key d853d3deb68be9d3dbbb8f37b7a81cd3
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:26.081075 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.069368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48213 (local address 127.27.46.254:33655)
0925 04:33:26.069489 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.069517 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.070153 (+ 636us) server_negotiation.cc:407] Connection header received
0925 04:33:26.071944 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.071947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.071998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.072079 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.073180 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.073763 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.074804 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.075019 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.075151 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.077747 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.077750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.077752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.077775 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.079129 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.079534 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.079536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.079538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.079606 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.079893 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.079895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.079896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.080061 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.080133 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.080709 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.080856 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:33:26.081985 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.069549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33655 (local address 127.27.46.193:48213)
0925 04:33:26.070011 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.070064 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.071679 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.072239 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.072245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.072564 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:26.073026 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.073039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.073881 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.073884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.074679 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.074687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.075185 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.075188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.075351 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.075896 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:26.075911 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:26.077580 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:26.079223 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.079228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.079238 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.079455 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.079698 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.079700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.079702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.079818 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.080144 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:26.080148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:26.080463 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.080650 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.080930 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":137,"threads_started":1}
I20260925 04:33:26.083498 28622 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33655
I20260925 04:33:26.083779 28622 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:26.084228 28622 heartbeater.cc:511] Master 127.27.46.254:33655 requested a full tablet report, sending...
I20260925 04:33:26.085681 28423 ts_manager.cc:194] Registered new tserver with Master: 7e23e0bd0c7d4567a77aab4fcb7f1e2d (127.27.46.193:44871)
I20260925 04:33:26.086803 28423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:48213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:26.096482 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:26.203454 28627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:26.203779 28627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:26.203862 28627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:26.207790 28627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:26.207885 28627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:26.207981 28627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:26.212870 28627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:26.213963 28627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:26.215091 28627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:26.225395 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.225395 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.225456 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:26.225682 28627 server_base.cc:1034] running on GCE node
I20260925 04:33:26.226245 28627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:26.226827 28627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:26.228019 28627 hybrid_clock.cc:648] HybridClock initialized: now 1790310806228010 us; error 27 us; skew 500 ppm
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:26.230762 28627 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:26.231666 28627 webserver.cc:460] Webserver started at http://127.27.46.194:33325/ using document root <none> and password file <none>
I20260925 04:33:26.232223 28627 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:26.232313 28627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:26.232568 28627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:26.234694 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "241d9f9f0787410289a5bbbba4576af0"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "caa86ec124b12abb5f4ddee809cb7c00"
server_key_iv: "6101246afe5cdc4ad63bbf1b700d2057"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.235293 28627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "241d9f9f0787410289a5bbbba4576af0"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "caa86ec124b12abb5f4ddee809cb7c00"
server_key_iv: "6101246afe5cdc4ad63bbf1b700d2057"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.239146 28627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:33:26.241487 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.242861 28627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:26.243007 28627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "241d9f9f0787410289a5bbbba4576af0"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "caa86ec124b12abb5f4ddee809cb7c00"
server_key_iv: "6101246afe5cdc4ad63bbf1b700d2057"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.243121 28627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:26.259828 28627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:26.263975 28627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:26.264181 28627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:26.265022 28627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:26.266116 28627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:26.266182 28627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.266244 28627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:26.266274 28627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.276252 28627 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:35501
I20260925 04:33:26.276262 28757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:35501 every 8 connection(s)
I20260925 04:33:26.277179 28627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:26.282478 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28627
I20260925 04:33:26.282600 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:26.282876 27835 external_mini_cluster.cc:1428] Setting key e08244eb0e9b00917567f4c223e1562a
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:26.291980 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.279102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60567 (local address 127.27.46.254:33655)
0925 04:33:26.279294 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.279298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.280106 (+ 808us) server_negotiation.cc:407] Connection header received
0925 04:33:26.281573 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.281576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.281624 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.281692 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.283051 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.283702 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.285137 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.285356 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.285440 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.288099 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.288106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.288112 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.288148 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.289884 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.290401 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.290406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.290410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.290506 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.290835 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.290839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.290840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.291088 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.291212 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.291669 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.291797 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:33:26.292784 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.279495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33655 (local address 127.27.46.194:60567)
0925 04:33:26.279938 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.279980 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.281286 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.281854 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.281860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.282281 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:26.282865 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.282880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.283839 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.283847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.285005 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.285019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.285542 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.285549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.285794 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.286493 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:26.286509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:26.287919 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:26.290029 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.290034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.290046 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.290297 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.290607 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.290609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.290610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.290737 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.291208 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:26.291213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:26.291428 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.291579 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.291818 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260925 04:33:26.294112 28758 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33655
I20260925 04:33:26.294325 28758 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:26.294755 28758 heartbeater.cc:511] Master 127.27.46.254:33655 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:33:26.295917 28423 ts_manager.cc:194] Registered new tserver with Master: 241d9f9f0787410289a5bbbba4576af0 (127.27.46.194:35501)
I20260925 04:33:26.296494 28423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:60567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:26.312765 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:26.425964 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:26.426213 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:26.426260 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:26.430423 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:26.430480 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:26.430572 28763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:26.435582 28763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:26.436641 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:26.437757 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:26.447361 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.447392 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:26.447320 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:26.447503 28763 server_base.cc:1034] running on GCE node
I20260925 04:33:26.447996 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:26.448611 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:26.449784 28763 hybrid_clock.cc:648] HybridClock initialized: now 1790310806449744 us; error 54 us; skew 500 ppm
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:26.452255 28763 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:26.453047 28763 webserver.cc:460] Webserver started at http://127.27.46.195:38347/ using document root <none> and password file <none>
I20260925 04:33:26.453615 28763 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:26.453704 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:26.453910 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:26.455860 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "e6d0604852fd47bebf346d87daa9ba8b"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "723ba38dfdc1fdbbd64c39d6b4c2eed0"
server_key_iv: "f1f0aebf17764215aef84d4d64506f6e"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.456411 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "e6d0604852fd47bebf346d87daa9ba8b"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "723ba38dfdc1fdbbd64c39d6b4c2eed0"
server_key_iv: "f1f0aebf17764215aef84d4d64506f6e"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.460057 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 04:33:26.462615 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.463819 28763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:26.463938 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "e6d0604852fd47bebf346d87daa9ba8b"
format_stamp: "Formatted at 2026-09-25 04:33:26 on dist-test-slave-r8r3"
server_key: "723ba38dfdc1fdbbd64c39d6b4c2eed0"
server_key_iv: "f1f0aebf17764215aef84d4d64506f6e"
server_key_version: "encryptionkey@0"
I20260925 04:33:26.464043 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:26.485039 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:26.487651 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:26.487850 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:26.488432 28763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:26.489233 28763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:26.489279 28763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.489346 28763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:26.489391 28763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:26.498030 28763 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:42405
I20260925 04:33:26.498062 28894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:42405 every 8 connection(s)
I20260925 04:33:26.498888 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:26.508944 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28763
I20260925 04:33:26.509049 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:26.509284 27835 external_mini_cluster.cc:1428] Setting key 581189a7d7ebd791fc6613fc9ee8c4fa
I20260925 04:33:26.511885 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.500447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:37029 (local address 127.27.46.254:33655)
0925 04:33:26.500579 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.500584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.501392 (+ 808us) server_negotiation.cc:407] Connection header received
0925 04:33:26.502455 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.502457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.502502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.502598 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.503663 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.504568 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.505650 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.505838 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.505922 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.508243 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.508254 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.508256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.508279 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.509719 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.510307 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.510309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.510311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.510383 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.510832 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.510835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.510837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.511053 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.511151 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.511681 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.511774 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:26.512633 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.500820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33655 (local address 127.27.46.195:37029)
0925 04:33:26.501263 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.501299 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.502274 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.502751 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.502758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.503081 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:26.503498 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.503510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.504679 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.504681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.505559 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.505567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.505980 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.505982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.506127 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.506657 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:26.506671 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:26.508073 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:26.509845 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.509851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.509863 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.510208 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.510507 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:26.510509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:26.510511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:26.510735 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:26.511152 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:26.511156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:26.511422 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.511623 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.511846 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260925 04:33:26.513782 28895 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33655
I20260925 04:33:26.513991 28895 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:26.514412 28895 heartbeater.cc:511] Master 127.27.46.254:33655 requested a full tablet report, sending...
I20260925 04:33:26.515347 28423 ts_manager.cc:194] Registered new tserver with Master: e6d0604852fd47bebf346d87daa9ba8b (127.27.46.195:42405)
I20260925 04:33:26.515851 28423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:37029
I20260925 04:33:26.522951 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:33:26.534080 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.527182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.27.46.254:33655)
0925 04:33:26.527294 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.527298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.527489 (+ 191us) server_negotiation.cc:407] Connection header received
0925 04:33:26.527585 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.527587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.527622 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.527687 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.528426 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.528920 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.529757 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.529975 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.530053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.531296 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.531299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.531301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.531322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.532715 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.533070 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.533072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.533073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.533138 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.533351 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.533353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.533354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.533603 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.533702 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.533877 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.533960 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:33:26.539774 28423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:26.542793 28423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:26.557169 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.551410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:42405 (local address 127.0.0.1:40232)
0925 04:33:26.552083 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.552132 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.552259 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.553068 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.553076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.553099 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:26.553237 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.553243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554413 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.554418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.555547 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.555557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.556607 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.556610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.556725 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.556768 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.556878 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.557007 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":71,"threads_started":1}
I20260925 04:33:26.557201 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.551567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.27.46.195:42405)
0925 04:33:26.552563 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.552582 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.552604 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:33:26.552679 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.552685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.552884 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.553021 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:26.553377 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554291 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.555664 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.556266 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.556389 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.556533 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.556936 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.557041 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":138,"threads_started":1}
I20260925 04:33:26.557696 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.551518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44871 (local address 127.0.0.1:60954)
0925 04:33:26.552398 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.552412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.552557 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.553220 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.553221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.553236 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:26.553375 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.553381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554787 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.554789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.556000 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.556010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.557261 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.557265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.557384 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.557409 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.557484 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.557572 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260925 04:33:26.558081 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.551873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.27.46.193:44871)
0925 04:33:26.552255 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.552270 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.552490 (+ 220us) server_negotiation.cc:407] Connection header received
0925 04:33:26.552696 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.552700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.552884 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.553010 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:26.553519 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554640 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.556146 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.556933 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.557027 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.557188 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.557546 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.557658 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":77,"threads_started":1}
I20260925 04:33:26.558235 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.552474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.27.46.194:35501)
0925 04:33:26.552814 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.552823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.553133 (+ 310us) server_negotiation.cc:407] Connection header received
0925 04:33:26.553210 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.553215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.553404 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.553611 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:26.553815 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554830 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.556325 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.556871 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.556974 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.557124 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.557916 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.558008 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":96,"threads_started":1}
I20260925 04:33:26.558504 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.552106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35501 (local address 127.0.0.1:52738)
0925 04:33:26.552921 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.552931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.553038 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.553581 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.553583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.553602 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:26.553698 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.553704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.554951 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.554954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.556220 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.556227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.557189 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.557195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.557549 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.557586 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.557655 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.557724 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":21,"thread_start_us":80,"threads_started":1}
I20260925 04:33:26.560629 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 29b775fca2dd4e5cae4adfa1c917162f (DEFAULT_TABLE table=test-table [id=3cbec52498d14a428435f621784cd54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:26.560690 28828 tablet_service.cc:1467] Processing CreateTablet for tablet 29b775fca2dd4e5cae4adfa1c917162f (DEFAULT_TABLE table=test-table [id=3cbec52498d14a428435f621784cd54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:26.560853 28691 tablet_service.cc:1467] Processing CreateTablet for tablet 29b775fca2dd4e5cae4adfa1c917162f (DEFAULT_TABLE table=test-table [id=3cbec52498d14a428435f621784cd54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:26.561721 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b775fca2dd4e5cae4adfa1c917162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:26.561735 28828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b775fca2dd4e5cae4adfa1c917162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:26.561734 28691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b775fca2dd4e5cae4adfa1c917162f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:26.568596 28912 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: Bootstrap starting.
I20260925 04:33:26.569386 28913 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Bootstrap starting.
I20260925 04:33:26.570464 28914 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: Bootstrap starting.
I20260925 04:33:26.570981 28912 tablet_bootstrap.cc:654] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:26.571574 28913 tablet_bootstrap.cc:654] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:26.572697 28914 tablet_bootstrap.cc:654] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:26.573448 28913 log.cc:824] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:26.573668 28912 log.cc:824] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:26.573683 28914 log.cc:824] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:26.575604 28914 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: No bootstrap required, opened a new log
I20260925 04:33:26.575667 28912 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: No bootstrap required, opened a new log
I20260925 04:33:26.575721 28913 tablet_bootstrap.cc:492] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: No bootstrap required, opened a new log
I20260925 04:33:26.575821 28914 ts_tablet_manager.cc:1397] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260925 04:33:26.575856 28913 ts_tablet_manager.cc:1397] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260925 04:33:26.575847 28912 ts_tablet_manager.cc:1397] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260925 04:33:26.578969 28914 raft_consensus.cc:348] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.579209 28914 raft_consensus.cc:374] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:26.579269 28914 raft_consensus.cc:729] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241d9f9f0787410289a5bbbba4576af0, State: Initialized, Role: FOLLOWER
I20260925 04:33:26.579741 28914 consensus_queue.cc:260] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.579996 28912 raft_consensus.cc:348] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.579996 28913 raft_consensus.cc:348] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.580222 28912 raft_consensus.cc:374] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:26.580221 28913 raft_consensus.cc:374] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:26.580276 28913 raft_consensus.cc:729] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d0604852fd47bebf346d87daa9ba8b, State: Initialized, Role: FOLLOWER
I20260925 04:33:26.580276 28912 raft_consensus.cc:729] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e23e0bd0c7d4567a77aab4fcb7f1e2d, State: Initialized, Role: FOLLOWER
I20260925 04:33:26.580708 28913 consensus_queue.cc:260] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.580719 28912 consensus_queue.cc:260] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.580785 28914 ts_tablet_manager.cc:1428] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260925 04:33:26.581089 28758 heartbeater.cc:503] Master 127.27.46.254:33655 was elected leader, sending a full tablet report...
I20260925 04:33:26.581363 28622 heartbeater.cc:503] Master 127.27.46.254:33655 was elected leader, sending a full tablet report...
I20260925 04:33:26.581681 28912 ts_tablet_manager.cc:1428] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260925 04:33:26.581773 28913 ts_tablet_manager.cc:1428] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 04:33:26.582048 28895 heartbeater.cc:503] Master 127.27.46.254:33655 was elected leader, sending a full tablet report...
I20260925 04:33:26.670094 28920 raft_consensus.cc:482] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:26.670333 28920 raft_consensus.cc:504] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.671504 28920 leader_election.cc:288] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e23e0bd0c7d4567a77aab4fcb7f1e2d (127.27.46.193:44871), 241d9f9f0787410289a5bbbba4576af0 (127.27.46.194:35501)
I20260925 04:33:26.675588 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.671757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44871 (local address 127.27.46.195:48715)
0925 04:33:26.671867 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.671884 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.671974 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.672285 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.672288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.672307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:26.672428 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.672434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.673373 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.673375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.674551 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.674562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.675275 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.675277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.675353 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.675373 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.675430 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.675482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":46}
I20260925 04:33:26.675657 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.671854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:48715 (local address 127.27.46.193:44871)
0925 04:33:26.671997 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.672004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.672023 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:33:26.672094 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.672097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.672152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.672248 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:26.672553 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.673267 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.674676 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.675163 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.675277 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.675327 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.675467 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.675531 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:33:26.676283 28576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b775fca2dd4e5cae4adfa1c917162f" candidate_uuid: "e6d0604852fd47bebf346d87daa9ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" is_pre_election: true
I20260925 04:33:26.676469 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.672158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:55163 (local address 127.27.46.194:35501)
0925 04:33:26.672354 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.672358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.672615 (+ 257us) server_negotiation.cc:407] Connection header received
0925 04:33:26.672702 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.672705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.672754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.672832 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:26.673070 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.673698 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.674989 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.675636 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.675750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.675822 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.676282 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.676366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 04:33:26.676513 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.672099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35501 (local address 127.27.46.195:55163)
0925 04:33:26.672522 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:26.672537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:26.672628 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:26.672848 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:26.672849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:26.672862 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:26.672966 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.672970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.673829 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.673832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.674881 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:26.674888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.675981 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.675987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:26.676136 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.676155 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:33:26.676264 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:33:26.676345 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":19,"thread_start_us":186,"threads_started":1}
I20260925 04:33:26.676649 28576 raft_consensus.cc:2454] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6d0604852fd47bebf346d87daa9ba8b in term 0.
I20260925 04:33:26.677165 28780 leader_election.cc:302] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e23e0bd0c7d4567a77aab4fcb7f1e2d, e6d0604852fd47bebf346d87daa9ba8b; no voters:
I20260925 04:33:26.677381 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b775fca2dd4e5cae4adfa1c917162f" candidate_uuid: "e6d0604852fd47bebf346d87daa9ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241d9f9f0787410289a5bbbba4576af0" is_pre_election: true
I20260925 04:33:26.677452 28920 raft_consensus.cc:2781] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:26.677563 28920 raft_consensus.cc:482] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:26.677620 28920 raft_consensus.cc:3037] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:26.677709 28711 raft_consensus.cc:2454] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6d0604852fd47bebf346d87daa9ba8b in term 0.
I20260925 04:33:26.678755 28920 raft_consensus.cc:504] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.679081 28920 leader_election.cc:288] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [CANDIDATE]: Term 1 election: Requested vote from peers 7e23e0bd0c7d4567a77aab4fcb7f1e2d (127.27.46.193:44871), 241d9f9f0787410289a5bbbba4576af0 (127.27.46.194:35501)
I20260925 04:33:26.679402 28576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b775fca2dd4e5cae4adfa1c917162f" candidate_uuid: "e6d0604852fd47bebf346d87daa9ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d"
I20260925 04:33:26.679519 28576 raft_consensus.cc:3037] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:26.679529 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b775fca2dd4e5cae4adfa1c917162f" candidate_uuid: "e6d0604852fd47bebf346d87daa9ba8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241d9f9f0787410289a5bbbba4576af0"
I20260925 04:33:26.679665 28711 raft_consensus.cc:3037] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:26.680527 28576 raft_consensus.cc:2454] T 29b775fca2dd4e5cae4adfa1c917162f P 7e23e0bd0c7d4567a77aab4fcb7f1e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6d0604852fd47bebf346d87daa9ba8b in term 1.
I20260925 04:33:26.680738 28711 raft_consensus.cc:2454] T 29b775fca2dd4e5cae4adfa1c917162f P 241d9f9f0787410289a5bbbba4576af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6d0604852fd47bebf346d87daa9ba8b in term 1.
I20260925 04:33:26.680886 28780 leader_election.cc:302] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e23e0bd0c7d4567a77aab4fcb7f1e2d, e6d0604852fd47bebf346d87daa9ba8b; no voters:
I20260925 04:33:26.681052 28920 raft_consensus.cc:2781] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:26.681283 28920 raft_consensus.cc:686] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [term 1 LEADER]: Becoming Leader. State: Replica: e6d0604852fd47bebf346d87daa9ba8b, State: Running, Role: LEADER
I20260925 04:33:26.681674 28920 consensus_queue.cc:237] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } }
I20260925 04:33:26.684945 28422 catalog_manager.cc:5495] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b reported cstate change: term changed from 0 to 1, leader changed from <none> to e6d0604852fd47bebf346d87daa9ba8b (127.27.46.195). New cstate: current_term: 1 leader_uuid: "e6d0604852fd47bebf346d87daa9ba8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6d0604852fd47bebf346d87daa9ba8b" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e23e0bd0c7d4567a77aab4fcb7f1e2d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "241d9f9f0787410289a5bbbba4576af0" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35501 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:26.708056 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.699744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.27.46.193:44871)
0925 04:33:26.699889 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.699893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.700101 (+ 208us) server_negotiation.cc:407] Connection header received
0925 04:33:26.700191 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.700194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.700230 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.700292 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.700943 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.701401 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.702226 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.702414 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.702488 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.704848 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.704870 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.704884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.704906 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.706585 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.706925 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.706930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.706931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.707004 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.707250 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.707253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.707254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.707472 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.707627 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.707759 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.707845 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:26.733588 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.725895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.27.46.193:44871)
0925 04:33:26.726027 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.726032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.726169 (+ 137us) server_negotiation.cc:407] Connection header received
0925 04:33:26.726259 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.726261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.726298 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.726359 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.726943 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.727463 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.728218 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.728425 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.728495 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.730705 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.730708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.730710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.730730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.732069 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.732430 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.732432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.732434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.732499 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.732726 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.732728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.732729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.732873 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.732964 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.733197 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.733292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260925 04:33:26.749820 28896 tablet.cc:2369] T 29b775fca2dd4e5cae4adfa1c917162f P e6d0604852fd47bebf346d87daa9ba8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:26.756855 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:26.749855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.27.46.193:44871)
0925 04:33:26.749982 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:26.749986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:26.750146 (+ 160us) server_negotiation.cc:407] Connection header received
0925 04:33:26.750229 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:26.750232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:26.750276 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:26.750358 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:26.750905 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.751343 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.752046 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:26.752225 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:26.752291 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:26.754412 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:26.754415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:26.754417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:26.754437 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:26.755575 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.755957 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.755959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.755960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.756026 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:26.756247 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:26.756249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:26.756250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:26.756389 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:26.756479 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:26.756662 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 04:33:26.756744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 04:33:26.758258 28534 tablet_service.cc:2908] Rejecting scan request for tablet 29b775fca2dd4e5cae4adfa1c917162f: Uninitialized: clean time has not yet been initialized
I20260925 04:33:26.760200 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28487
I20260925 04:33:26.766880 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28627
I20260925 04:33:26.772739 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28763
I20260925 04:33:26.778544 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28390
2026-09-25T04:33:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2301 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28938](info): set up 2 sockets
Sep 25 04:33:26 dist-test-slave-r8r3 krb5kdc[28938](info): commencing operation
krb5kdc: starting...
W20260925 04:33:27.801586 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:27 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:27.919674 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44673
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:28.025707 28951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:28.025959 28951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:28.026100 28951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:28.030014 28951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:28.030059 28951 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:33:28.030076 28951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:28.030093 28951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:28.030157 28951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:28.030177 28951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:28.035131 28951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:44673
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:28.036095 28951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:28.037062 28951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:28.045413 28956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.045439 28957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.045462 28959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:28.045579 28951 server_base.cc:1034] running on GCE node
I20260925 04:33:28.046162 28951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:28.047140 28951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:28.048316 28951 hybrid_clock.cc:648] HybridClock initialized: now 1790310808048290 us; error 38 us; skew 500 ppm
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:28.056648 28951 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:28.057535 28951 webserver.cc:460] Webserver started at http://127.27.46.254:37463/ using document root <none> and password file <none>
I20260925 04:33:28.058054 28951 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:28.058110 28951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:28.058262 28951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:28.060217 28951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "80131fa4772041d0ba1ce55fc987fd25"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "cbb4cfd4b57bacd55bf25e4fb375cdb4"
server_key_iv: "96648ea73239eda9cd706405c7223adc"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.060704 28951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "80131fa4772041d0ba1ce55fc987fd25"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "cbb4cfd4b57bacd55bf25e4fb375cdb4"
server_key_iv: "96648ea73239eda9cd706405c7223adc"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.064132 28951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 04:33:28.066385 28966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.067548 28951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:28.067679 28951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "80131fa4772041d0ba1ce55fc987fd25"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "cbb4cfd4b57bacd55bf25e4fb375cdb4"
server_key_iv: "96648ea73239eda9cd706405c7223adc"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.067775 28951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:28.080928 28951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:28.083537 28951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:28.083766 28951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:28.091462 28951 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:44673
I20260925 04:33:28.091490 29028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:44673 every 8 connection(s)
I20260925 04:33:28.092330 28951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:28.094906 29029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:28.095178 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 28951
I20260925 04:33:28.095294 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:28.095542 27835 external_mini_cluster.cc:1428] Setting key e19ee5fe9f5186ff71d87465995fe79e
I20260925 04:33:28.101842 29029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Bootstrap starting.
I20260925 04:33:28.104274 29029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Neither blocks nor log segments found. Creating new log.
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:28.105173 29029 log.cc:824] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:28.107071 29029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: No bootstrap required, opened a new log
I20260925 04:33:28.110074 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.096920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.27.46.254:44673)
0925 04:33:28.097328 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.097346 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.097388 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:33:28.098178 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.098200 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.098624 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.098967 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.099623 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.101130 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.101988 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.102436 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.102662 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.105341 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.105346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.105357 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.105380 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.107460 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.107938 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.107942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.107947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.108050 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.108330 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.108332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.108333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.108689 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.108822 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.109119 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.109393 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260925 04:33:28.110410 29029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } }
I20260925 04:33:28.110641 29029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:28.110709 29029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80131fa4772041d0ba1ce55fc987fd25, State: Initialized, Role: FOLLOWER
I20260925 04:33:28.111208 29029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } }
I20260925 04:33:28.111348 29029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:28.111398 29029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:28.111481 29029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:28.112344 29029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } }
I20260925 04:33:28.112727 29029 leader_election.cc:302] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80131fa4772041d0ba1ce55fc987fd25; no voters:
I20260925 04:33:28.113098 29029 leader_election.cc:288] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:28.113194 29034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:28.113368 29034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [term 1 LEADER]: Becoming Leader. State: Replica: 80131fa4772041d0ba1ce55fc987fd25, State: Running, Role: LEADER
I20260925 04:33:28.113682 29034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } }
I20260925 04:33:28.114310 29029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:28.115134 29035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80131fa4772041d0ba1ce55fc987fd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } } }
I20260925 04:33:28.115248 29035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:28.115592 29043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:28.115623 29036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80131fa4772041d0ba1ce55fc987fd25. Latest consensus state: current_term: 1 leader_uuid: "80131fa4772041d0ba1ce55fc987fd25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80131fa4772041d0ba1ce55fc987fd25" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44673 } } }
I20260925 04:33:28.116237 29036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:28.118270 29043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:28.122921 29043 catalog_manager.cc:1347] Generated new cluster ID: caee408f445c4a60b45078b9be6883b5
I20260925 04:33:28.122978 29043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:28.143989 29043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:28.145185 29043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:28.160334 29043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Generated new TSK 0
I20260925 04:33:28.161051 29043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:28.201100 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44673
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:28.321843 29056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:28.322166 29056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:28.322244 29056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:28.326344 29056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:28.326434 29056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:28.326506 29056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:28.326587 29056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:28.331696 29056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:28.332775 29056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:28.333824 29056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:28.343384 29063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.343389 29062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:28.343739 29056 server_base.cc:1034] running on GCE node
W20260925 04:33:28.343389 29065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:28.344190 29056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:28.344777 29056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:28.345992 29056 hybrid_clock.cc:648] HybridClock initialized: now 1790310808345978 us; error 25 us; skew 500 ppm
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:28.348670 29056 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:28.349634 29056 webserver.cc:460] Webserver started at http://127.27.46.193:41513/ using document root <none> and password file <none>
I20260925 04:33:28.350178 29056 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:28.350267 29056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:28.350497 29056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:28.352470 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "a08f97722694af1b9c470aa24f29d292"
server_key_iv: "4ea13bd17e5ea54cf0c4fe54971bc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.353065 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "a08f97722694af1b9c470aa24f29d292"
server_key_iv: "4ea13bd17e5ea54cf0c4fe54971bc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.356928 29056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:33:28.359447 29072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.360637 29056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:28.360781 29056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "a08f97722694af1b9c470aa24f29d292"
server_key_iv: "4ea13bd17e5ea54cf0c4fe54971bc043"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.360911 29056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:28.389919 29056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:28.392784 29056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:28.392995 29056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:28.394501 29056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:28.394577 29056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.394654 29056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:28.394697 29056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:28.405404 29056 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:34339
I20260925 04:33:28.405544 29188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:34339 every 8 connection(s)
I20260925 04:33:28.406584 29056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:28.408375 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29056
I20260925 04:33:28.408491 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:28.408782 27835 external_mini_cluster.cc:1428] Setting key 8aa5bd580cbe8531b66d20886503f8b8
I20260925 04:33:28.409428 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.395618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:38685 (local address 127.27.46.254:44673)
0925 04:33:28.395785 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.395791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.396740 (+ 949us) server_negotiation.cc:407] Connection header received
0925 04:33:28.398152 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.398156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.398223 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.398331 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.400013 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.400637 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.402527 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.402754 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.402839 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.405682 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.405686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.405689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.405717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.407275 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.407860 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.407863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.407864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.407946 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.408225 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.408230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.408231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.408495 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.408600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.409129 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.409255 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:33:28.410306 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.396027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.193:38685)
0925 04:33:28.396546 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.396605 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.397892 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.398547 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.398556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.399054 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.399836 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.399856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.400782 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.400788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.402402 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.402413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.402930 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.402935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.403152 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.403880 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.403900 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.405492 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.407414 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.407419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.407434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.407754 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.408009 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.408010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.408012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.408126 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.408595 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.408601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.408859 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.409071 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.409378 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":129,"threads_started":1}
I20260925 04:33:28.411720 29189 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44673
I20260925 04:33:28.411887 29189 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:28.412462 29189 heartbeater.cc:511] Master 127.27.46.254:44673 requested a full tablet report, sending...
I20260925 04:33:28.414098 28982 ts_manager.cc:194] Registered new tserver with Master: 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339)
I20260925 04:33:28.415350 28982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:38685
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:28.424561 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.414173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:36831 (local address 127.27.46.254:44673)
0925 04:33:28.414280 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.414284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.417768 (+ 3484us) server_negotiation.cc:407] Connection header received
0925 04:33:28.417892 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.417896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.417941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.418032 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.418456 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.418948 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.419791 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.420133 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.420219 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.421315 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.421318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.421321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.421351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.422975 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.423376 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.423380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.423382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.423457 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.423758 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.423763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.423765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.423941 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.424049 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.424300 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.424409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:33:28.424578 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.413790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.193:36831)
0925 04:33:28.417683 (+ 3893us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.417695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.417805 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.418040 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.418043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.418236 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.418324 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.418329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.419077 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.419080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.419649 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.419657 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.420237 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.420239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.420306 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.420857 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.420870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.421170 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.423088 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.423091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.423093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.423280 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.423547 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.423551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.423553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.423646 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.424039 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.424042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.424138 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.424246 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.424396 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:28.438516 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44673
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:28.544828 29199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:28.545109 29199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:28.545182 29199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:28.549081 29199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:28.549180 29199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:28.549248 29199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:28.549322 29199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:28.554095 29199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:28.555053 29199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:28.555998 29199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:28.564886 29206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.564916 29205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.564895 29208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:28.565224 29199 server_base.cc:1034] running on GCE node
I20260925 04:33:28.565711 29199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:28.566521 29199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:28.567752 29199 hybrid_clock.cc:648] HybridClock initialized: now 1790310808567731 us; error 38 us; skew 500 ppm
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:28.570370 29199 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:28.571187 29199 webserver.cc:460] Webserver started at http://127.27.46.194:37125/ using document root <none> and password file <none>
I20260925 04:33:28.571722 29199 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:28.571808 29199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:28.572031 29199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:28.573972 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "dd4187c02f0342a09bd79395cf400905"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "4f3f2da5beed23b2cc14dd8e600f340c"
server_key_iv: "4b8378cf01769fc83ae45e7df5e29b57"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.574527 29199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "dd4187c02f0342a09bd79395cf400905"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "4f3f2da5beed23b2cc14dd8e600f340c"
server_key_iv: "4b8378cf01769fc83ae45e7df5e29b57"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.578130 29199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:28.580366 29216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.581496 29199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:28.581667 29199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "dd4187c02f0342a09bd79395cf400905"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "4f3f2da5beed23b2cc14dd8e600f340c"
server_key_iv: "4b8378cf01769fc83ae45e7df5e29b57"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.581764 29199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:28.591342 29199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:28.593771 29199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:28.593953 29199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:28.595192 29199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:28.595268 29199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.595350 29199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:28.595395 29199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:28.605734 29199 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:38505
I20260925 04:33:28.605798 29333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:38505 every 8 connection(s)
I20260925 04:33:28.606585 29199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:28.608593 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.595979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60301 (local address 127.27.46.254:44673)
0925 04:33:28.596121 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.596126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.596898 (+ 772us) server_negotiation.cc:407] Connection header received
0925 04:33:28.598058 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.598062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.598125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.598233 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.599560 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.600077 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.601741 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.602122 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.602238 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.604994 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.605000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.605006 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.605040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.606495 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.607024 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.607028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.607030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.607118 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.607502 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.607507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.607508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.607698 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.607803 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.608279 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.608401 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:33:28.609251 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.596328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.194:60301)
0925 04:33:28.596723 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.596768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.597812 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.598453 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.598464 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.598870 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.599375 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.599393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.600207 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.600210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.601584 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.601595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.602318 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.602322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.602526 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.603167 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.603182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.604813 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.606639 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.606644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.606657 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.606905 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.607217 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.607221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.607223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.607403 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.607816 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.607824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.608053 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.608237 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.608532 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260925 04:33:28.610369 29334 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44673
I20260925 04:33:28.610512 29334 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:28.610970 29334 heartbeater.cc:511] Master 127.27.46.254:44673 requested a full tablet report, sending...
I20260925 04:33:28.611934 28982 ts_manager.cc:194] Registered new tserver with Master: dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505)
I20260925 04:33:28.612689 28982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:60301
I20260925 04:33:28.614761 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29199
I20260925 04:33:28.614856 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:28.615214 27835 external_mini_cluster.cc:1428] Setting key 6515078f94c70998e63ef7a44a251e26
I20260925 04:33:28.619663 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.612514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:50897 (local address 127.27.46.254:44673)
0925 04:33:28.612641 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.612645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.612795 (+ 150us) server_negotiation.cc:407] Connection header received
0925 04:33:28.612897 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.612900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.612963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.613040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.613572 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.614097 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.615027 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.615311 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.615390 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.616609 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.616612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.616613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.616633 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.618020 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.618461 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.618464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.618466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.618546 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.618827 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.618831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.618833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.619049 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.619169 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.619420 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.619523 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:28.619674 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.612403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.194:50897)
0925 04:33:28.612702 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.612718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.612815 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.613144 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.613147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.613332 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.613421 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.613427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.614202 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.614205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.614896 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.614906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.615421 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.615425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.615509 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.615995 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.616006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.616332 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.618160 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.618163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.618165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.618359 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.618641 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.618645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.618647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.618734 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.619161 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.619164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.619258 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.619364 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.619500 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:28.640466 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44673
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:28.748126 29343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:28.748355 29343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:28.748409 29343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:28.752306 29343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:28.752358 29343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:28.752395 29343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:28.752426 29343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:28.757471 29343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:28.758494 29343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:28.759529 29343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:28.768914 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.768888 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:28.768909 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:28.769167 29343 server_base.cc:1034] running on GCE node
I20260925 04:33:28.769695 29343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:28.770210 29343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:28.771390 29343 hybrid_clock.cc:648] HybridClock initialized: now 1790310808771380 us; error 42 us; skew 500 ppm
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:28.774006 29343 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:28.774864 29343 webserver.cc:460] Webserver started at http://127.27.46.195:37797/ using document root <none> and password file <none>
I20260925 04:33:28.775455 29343 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:28.775561 29343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:28.775780 29343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:28.777760 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "045f0859cd2a4a36851a659fe401f685"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "fad683e35f8c86787898dae0f8a71436"
server_key_iv: "782897bb8b7ef8df544ba2d33cb31e70"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.778306 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "045f0859cd2a4a36851a659fe401f685"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "fad683e35f8c86787898dae0f8a71436"
server_key_iv: "782897bb8b7ef8df544ba2d33cb31e70"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.781927 29343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:28.784157 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.785598 29343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:28.785725 29343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "045f0859cd2a4a36851a659fe401f685"
format_stamp: "Formatted at 2026-09-25 04:33:28 on dist-test-slave-r8r3"
server_key: "fad683e35f8c86787898dae0f8a71436"
server_key_iv: "782897bb8b7ef8df544ba2d33cb31e70"
server_key_version: "encryptionkey@0"
I20260925 04:33:28.785861 29343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:28.816612 29343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:28.820152 29343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:28.820387 29343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:28.822007 29343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:28.822083 29343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:28.822157 29343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:28.822217 29343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:28.834174 29343 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:43747
I20260925 04:33:28.834245 29476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:43747 every 8 connection(s)
I20260925 04:33:28.835253 29343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:28.836388 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29343
I20260925 04:33:28.836512 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:28.836580 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.822691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:56835 (local address 127.27.46.254:44673)
0925 04:33:28.822875 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.822882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.824188 (+ 1306us) server_negotiation.cc:407] Connection header received
0925 04:33:28.825335 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.825339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.825395 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.825470 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.826767 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.827677 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.828852 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.829081 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.829197 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.832338 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.832342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.832345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.832374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.834190 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.834760 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.834763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.834764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.834834 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.835185 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.835187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.835188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.835386 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.835469 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.836292 (+ 823us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.836403 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:33:28.836817 27835 external_mini_cluster.cc:1428] Setting key d0fca9c975a6ac5252b2f0cad28d3e1c
I20260925 04:33:28.837270 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.823523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.195:56835)
0925 04:33:28.824001 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.824064 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.825135 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.825661 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.825668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.825987 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.826579 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.826600 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.827844 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.827848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.828723 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.828732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.829249 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.829252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.829443 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.830126 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.830141 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.832167 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.834327 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.834333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.834346 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.834658 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.834934 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.834937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.834939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.835093 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.835479 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.835484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.835777 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.836012 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.836338 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":154,"threads_started":1}
I20260925 04:33:28.838565 29477 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44673
I20260925 04:33:28.838706 29477 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:28.839226 29477 heartbeater.cc:511] Master 127.27.46.254:44673 requested a full tablet report, sending...
I20260925 04:33:28.840178 28982 ts_manager.cc:194] Registered new tserver with Master: 045f0859cd2a4a36851a659fe401f685 (127.27.46.195:43747)
I20260925 04:33:28.840899 28982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:56835
I20260925 04:33:28.847854 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.840670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:47507 (local address 127.27.46.254:44673)
0925 04:33:28.840879 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.840883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.840939 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:33:28.841096 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.841099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.841187 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.841273 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.841762 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.842479 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.843328 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.843621 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.843751 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.844918 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.844922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.844924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.844954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.846107 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.846588 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.846593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.846595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.846681 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.846978 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.846983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.846985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.847159 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.847290 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.847578 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.847688 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260925 04:33:28.847882 29483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.840597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.27.46.195:47507)
0925 04:33:28.840865 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.840882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.840997 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.841301 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.841304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.841532 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:28.841647 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.841653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.842624 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.842628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.843215 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.843223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.843756 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.843760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.843858 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.844387 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:28.844404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:28.844740 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:28.846217 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.846223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.846225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.846454 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.846784 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:28.846788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:28.846790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:28.846881 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:28.847293 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:28.847297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:28.847402 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.847522 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.847664 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260925 04:33:28.850781 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:33:28 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:28.875363 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.867506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.27.46.254:44673)
0925 04:33:28.867625 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.867629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.867640 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:33:28.867678 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.867680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.867716 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.867803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:28.868480 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.868992 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.869934 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.870155 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.870279 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.872570 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:28.872574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:28.872577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:28.872601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:28.873769 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.874165 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.874168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.874170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.874245 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:28.874531 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:28.874537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:28.874542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:28.874709 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:28.874844 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.875090 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.875203 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:28.878028 28982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:28.880999 28982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:28.895452 29497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.889971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:43747 (local address 127.0.0.1:54606)
0925 04:33:28.890293 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.890308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.890378 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.891205 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.891208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.891231 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:28.891526 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.891534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892991 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.892995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.893988 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.894000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.894852 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.894855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.894969 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895009 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.895160 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.895276 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":142,"spinlock_wait_cycles":10112,"thread_start_us":97,"threads_started":1}
I20260925 04:33:28.895622 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.889407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.27.46.193:34339)
0925 04:33:28.890610 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.890627 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.890651 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:33:28.890725 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.890730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.890948 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.891181 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:28.891643 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892840 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.894268 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.894988 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.895141 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895292 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.895385 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.895477 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":147,"threads_started":1}
I20260925 04:33:28.895622 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.890405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.27.46.195:43747)
0925 04:33:28.890825 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.890835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.890851 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:33:28.890911 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.890915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.891068 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.891295 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:28.891643 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892832 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.894134 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.894722 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.894851 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895025 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.895386 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.895478 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":122,"threads_started":1}
I20260925 04:33:28.895887 29494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.889420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38505 (local address 127.0.0.1:46172)
0925 04:33:28.889892 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.889940 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.890068 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.891195 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.891198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.891223 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:28.891670 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.891674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892644 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.892649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.894014 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.894026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.895501 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.895505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.895614 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895636 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.895701 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.895779 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":264,"spinlock_wait_cycles":7808,"thread_start_us":135,"threads_started":1}
I20260925 04:33:28.896273 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.889181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34339 (local address 127.0.0.1:40264)
0925 04:33:28.889893 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:28.889941 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:28.890067 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:28.891171 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:28.891177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:28.891198 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:28.891495 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.891505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892952 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.892956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.894146 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:28.894154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.895167 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.895170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:28.895251 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895271 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:33:28.895339 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:33:28.895401 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":132,"spinlock_wait_cycles":5760,"thread_start_us":93,"threads_started":1}
I20260925 04:33:28.898444 29411 tablet_service.cc:1467] Processing CreateTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 (DEFAULT_TABLE table=test-table [id=61ae72403e194cb68ffb8074a00b6985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:28.898702 29122 tablet_service.cc:1467] Processing CreateTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 (DEFAULT_TABLE table=test-table [id=61ae72403e194cb68ffb8074a00b6985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:28.899451 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116c6267eb14298a354bf79f7ec2fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:28.899629 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116c6267eb14298a354bf79f7ec2fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:28.905766 29500 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Bootstrap starting.
I20260925 04:33:28.905908 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:28.890110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.27.46.194:38505)
0925 04:33:28.890690 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:28.890704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:28.890727 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:33:28.890803 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:28.890808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:28.891015 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:28.891403 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:28.891787 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.892530 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.894156 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:28.894988 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:28.895140 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:28.895292 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:28.905616 (+ 10324us) server_negotiation.cc:299] Negotiation successful
0925 04:33:28.905745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":113,"threads_started":1}
I20260925 04:33:28.906585 29501 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Bootstrap starting.
I20260925 04:33:28.908201 29500 tablet_bootstrap.cc:654] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:28.908416 29501 tablet_bootstrap.cc:654] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:28.908655 29267 tablet_service.cc:1467] Processing CreateTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 (DEFAULT_TABLE table=test-table [id=61ae72403e194cb68ffb8074a00b6985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:28.909067 29500 log.cc:824] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:28.909086 29501 log.cc:824] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:28.909749 29267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d116c6267eb14298a354bf79f7ec2fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:28.911106 29500 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: No bootstrap required, opened a new log
I20260925 04:33:28.911159 29501 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: No bootstrap required, opened a new log
I20260925 04:33:28.911312 29500 ts_tablet_manager.cc:1397] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:33:28.911363 29501 ts_tablet_manager.cc:1397] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:33:28.914546 29501 raft_consensus.cc:348] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.914817 29501 raft_consensus.cc:374] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:28.914875 29501 raft_consensus.cc:729] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab545520a2b42e3a8fc74dcaf8a735a, State: Initialized, Role: FOLLOWER
I20260925 04:33:28.915337 29501 consensus_queue.cc:260] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.915527 29500 raft_consensus.cc:348] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.915758 29500 raft_consensus.cc:374] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:28.915825 29500 raft_consensus.cc:729] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f0859cd2a4a36851a659fe401f685, State: Initialized, Role: FOLLOWER
I20260925 04:33:28.916275 29500 consensus_queue.cc:260] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.916360 29501 ts_tablet_manager.cc:1428] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260925 04:33:28.916735 29189 heartbeater.cc:503] Master 127.27.46.254:44673 was elected leader, sending a full tablet report...
I20260925 04:33:28.916951 29477 heartbeater.cc:503] Master 127.27.46.254:44673 was elected leader, sending a full tablet report...
I20260925 04:33:28.917193 29500 ts_tablet_manager.cc:1428] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:33:28.918490 29506 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Bootstrap starting.
I20260925 04:33:28.920661 29506 tablet_bootstrap.cc:654] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:28.921597 29506 log.cc:824] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:28.923192 29506 tablet_bootstrap.cc:492] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: No bootstrap required, opened a new log
I20260925 04:33:28.923377 29506 ts_tablet_manager.cc:1397] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:33:28.926285 29506 raft_consensus.cc:348] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.926468 29506 raft_consensus.cc:374] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:28.926553 29506 raft_consensus.cc:729] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4187c02f0342a09bd79395cf400905, State: Initialized, Role: FOLLOWER
I20260925 04:33:28.926916 29506 consensus_queue.cc:260] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:28.927587 29334 heartbeater.cc:503] Master 127.27.46.254:44673 was elected leader, sending a full tablet report...
I20260925 04:33:28.927835 29506 ts_tablet_manager.cc:1428] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 04:33:29.065604 29508 raft_consensus.cc:482] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:29.065946 29508 raft_consensus.cc:504] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.067183 29508 leader_election.cc:288] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), 045f0859cd2a4a36851a659fe401f685 (127.27.46.195:43747)
I20260925 04:33:29.071309 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.067827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:48231 (local address 127.27.46.195:43747)
0925 04:33:29.067961 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.067964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.068067 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:33:29.068178 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.068181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.068225 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.068294 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:29.068577 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.069121 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.070207 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.070712 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.070846 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.070904 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.071136 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.071210 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:33:29.071529 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.067543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:38569 (local address 127.27.46.193:34339)
0925 04:33:29.067682 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.067686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.067807 (+ 121us) server_negotiation.cc:407] Connection header received
0925 04:33:29.067961 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.067965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.068029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.068110 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:29.068414 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.068992 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.070597 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.070986 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.071112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.071165 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.071342 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.071408 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:33:29.071535 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.067464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34339 (local address 127.27.46.194:38569)
0925 04:33:29.067701 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.067718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.067824 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.068141 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.068144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.068165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:29.068296 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.068303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.069114 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.069119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.070464 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.070473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.071100 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.071104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.071222 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.071239 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.071327 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.071396 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 04:33:29.071799 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.067464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:43747 (local address 127.27.46.194:48231)
0925 04:33:29.067943 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.067958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.068093 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.068309 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.068311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.068324 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:29.068464 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.068470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.069234 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.069238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.070079 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.070088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.070839 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.070843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.070949 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.070977 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.071509 (+ 532us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.071605 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":60,"thread_start_us":132,"threads_started":1}
I20260925 04:33:29.072232 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116c6267eb14298a354bf79f7ec2fd6" candidate_uuid: "dd4187c02f0342a09bd79395cf400905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" is_pre_election: true
I20260925 04:33:29.072562 29143 raft_consensus.cc:2454] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4187c02f0342a09bd79395cf400905 in term 0.
I20260925 04:33:29.072566 29431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116c6267eb14298a354bf79f7ec2fd6" candidate_uuid: "dd4187c02f0342a09bd79395cf400905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f0859cd2a4a36851a659fe401f685" is_pre_election: true
I20260925 04:33:29.072813 29431 raft_consensus.cc:2454] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4187c02f0342a09bd79395cf400905 in term 0.
I20260925 04:33:29.073046 29218 leader_election.cc:302] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab545520a2b42e3a8fc74dcaf8a735a, dd4187c02f0342a09bd79395cf400905; no voters:
I20260925 04:33:29.073325 29508 raft_consensus.cc:2781] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:29.073419 29508 raft_consensus.cc:482] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:29.073474 29508 raft_consensus.cc:3037] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.074457 29508 raft_consensus.cc:504] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.074781 29508 leader_election.cc:288] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), 045f0859cd2a4a36851a659fe401f685 (127.27.46.195:43747)
I20260925 04:33:29.075120 29431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116c6267eb14298a354bf79f7ec2fd6" candidate_uuid: "dd4187c02f0342a09bd79395cf400905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f0859cd2a4a36851a659fe401f685"
I20260925 04:33:29.075170 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d116c6267eb14298a354bf79f7ec2fd6" candidate_uuid: "dd4187c02f0342a09bd79395cf400905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
I20260925 04:33:29.075243 29431 raft_consensus.cc:3037] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.075299 29143 raft_consensus.cc:3037] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.076553 29143 raft_consensus.cc:2454] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4187c02f0342a09bd79395cf400905 in term 1.
I20260925 04:33:29.076858 29218 leader_election.cc:302] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab545520a2b42e3a8fc74dcaf8a735a, dd4187c02f0342a09bd79395cf400905; no voters:
I20260925 04:33:29.077018 29508 raft_consensus.cc:2781] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:29.077135 29431 raft_consensus.cc:2454] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4187c02f0342a09bd79395cf400905 in term 1.
I20260925 04:33:29.077250 29508 raft_consensus.cc:686] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 1 LEADER]: Becoming Leader. State: Replica: dd4187c02f0342a09bd79395cf400905, State: Running, Role: LEADER
I20260925 04:33:29.077638 29508 consensus_queue.cc:237] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.080687 28983 catalog_manager.cc:5495] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd4187c02f0342a09bd79395cf400905 (127.27.46.194). New cstate: current_term: 1 leader_uuid: "dd4187c02f0342a09bd79395cf400905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } health_report { overall_health: UNKNOWN } } }
W20260925 04:33:29.086136 29478 tablet.cc:2369] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:33:29.108130 29335 tablet.cc:2369] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:29.133697 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.130485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.27.46.194:38505)
0925 04:33:29.130605 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.130609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.130622 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:33:29.130800 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.130802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.130853 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.130933 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:29.131157 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.131634 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.132413 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.132625 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.132698 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.132970 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.133336 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.133415 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.133486 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.133593 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:33:29.139391 29431 raft_consensus.cc:1261] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Refusing update from remote peer dd4187c02f0342a09bd79395cf400905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.139391 29143 raft_consensus.cc:1261] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Refusing update from remote peer dd4187c02f0342a09bd79395cf400905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.140013 29508 consensus_queue.cc:1035] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.140245 29510 consensus_queue.cc:1035] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.147152 29513 mvcc.cc:205] Tried to move back new op lower bound from 7333113074227384320 to 7333113073984806912. Current Snapshot: MvccSnapshot[applied={T|T < 7333113074227384320}]
I20260925 04:33:29.148276 29515 mvcc.cc:205] Tried to move back new op lower bound from 7333113074227384320 to 7333113073984806912. Current Snapshot: MvccSnapshot[applied={T|T < 7333113074227384320}]
I20260925 04:33:29.157955 28983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:29.158133 28983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:29.160420 28983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Sending DeleteTablet for 3 replicas of tablet d116c6267eb14298a354bf79f7ec2fd6
I20260925 04:33:29.161100 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40264
I20260925 04:33:29.161180 29411 tablet_service.cc:1514] Processing DeleteTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:54606
I20260925 04:33:29.161258 29267 tablet_service.cc:1514] Processing DeleteTablet for tablet d116c6267eb14298a354bf79f7ec2fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:46172
I20260925 04:33:29.161724 29525 tablet_replica.cc:330] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: stopping tablet replica
I20260925 04:33:29.161991 29525 raft_consensus.cc:2229] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:29.162191 29523 tablet_replica.cc:330] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: stopping tablet replica
I20260925 04:33:29.162340 29525 raft_consensus.cc:2258] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.162469 29523 raft_consensus.cc:2229] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:29.162755 29523 raft_consensus.cc:2258] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.162813 28983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:33:29.163082 29524 tablet_replica.cc:330] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: stopping tablet replica
W20260925 04:33:29.163206 28983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:29.163341 29524 raft_consensus.cc:2229] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:29.163616 29524 raft_consensus.cc:2258] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.163703 29525 ts_tablet_manager.cc:1905] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.165248 29524 ts_tablet_manager.cc:1905] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.166251 29523 ts_tablet_manager.cc:1905] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.168118 29524 ts_tablet_manager.cc:1918] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:29.168207 29524 log.cc:1196] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/d116c6267eb14298a354bf79f7ec2fd6
I20260925 04:33:29.168519 29524 ts_tablet_manager.cc:1939] T d116c6267eb14298a354bf79f7ec2fd6 P 045f0859cd2a4a36851a659fe401f685: Deleting consensus metadata
I20260925 04:33:29.169185 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (DEFAULT_TABLE table=test-table [id=8682bf03403e4111b8ec478ad6a9697f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:29.169317 29523 ts_tablet_manager.cc:1918] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:29.169389 29523 log.cc:1196] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/d116c6267eb14298a354bf79f7ec2fd6
I20260925 04:33:29.169412 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fdaf6e7e94fec9d2c11f36c7acf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.169684 29523 ts_tablet_manager.cc:1939] T d116c6267eb14298a354bf79f7ec2fd6 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting consensus metadata
I20260925 04:33:29.170332 29525 ts_tablet_manager.cc:1918] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:29.170405 29525 log.cc:1196] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/d116c6267eb14298a354bf79f7ec2fd6
I20260925 04:33:29.170729 29525 ts_tablet_manager.cc:1939] T d116c6267eb14298a354bf79f7ec2fd6 P dd4187c02f0342a09bd79395cf400905: Deleting consensus metadata
I20260925 04:33:29.171378 29501 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Bootstrap starting.
I20260925 04:33:29.172011 29267 tablet_service.cc:1467] Processing CreateTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (DEFAULT_TABLE table=test-table [id=8682bf03403e4111b8ec478ad6a9697f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:29.172254 29267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fdaf6e7e94fec9d2c11f36c7acf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.172513 29501 tablet_bootstrap.cc:654] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.172829 28968 catalog_manager.cc:4841] TS 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339): tablet d116c6267eb14298a354bf79f7ec2fd6 (table test-table [id=61ae72403e194cb68ffb8074a00b6985]) successfully deleted
I20260925 04:33:29.173123 28967 catalog_manager.cc:4841] TS 045f0859cd2a4a36851a659fe401f685 (127.27.46.195:43747): tablet d116c6267eb14298a354bf79f7ec2fd6 (table test-table [id=61ae72403e194cb68ffb8074a00b6985]) successfully deleted
I20260925 04:33:29.173236 28967 catalog_manager.cc:4841] TS dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505): tablet d116c6267eb14298a354bf79f7ec2fd6 (table test-table [id=61ae72403e194cb68ffb8074a00b6985]) successfully deleted
I20260925 04:33:29.174031 29411 tablet_service.cc:1467] Processing CreateTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (DEFAULT_TABLE table=test-table [id=8682bf03403e4111b8ec478ad6a9697f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:29.174255 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043fdaf6e7e94fec9d2c11f36c7acf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.174564 29506 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Bootstrap starting.
I20260925 04:33:29.175567 29506 tablet_bootstrap.cc:654] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.176642 29500 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Bootstrap starting.
I20260925 04:33:29.176782 29506 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: No bootstrap required, opened a new log
I20260925 04:33:29.176857 29506 ts_tablet_manager.cc:1397] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:29.177348 29506 raft_consensus.cc:348] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.177457 29506 raft_consensus.cc:374] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.177479 29501 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: No bootstrap required, opened a new log
I20260925 04:33:29.177492 29506 raft_consensus.cc:729] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4187c02f0342a09bd79395cf400905, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.177592 29501 ts_tablet_manager.cc:1397] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 04:33:29.177642 29500 tablet_bootstrap.cc:654] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.177683 29506 consensus_queue.cc:260] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.178249 29501 raft_consensus.cc:348] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.178345 29501 raft_consensus.cc:374] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.178375 29501 raft_consensus.cc:729] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab545520a2b42e3a8fc74dcaf8a735a, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.178486 29501 consensus_queue.cc:260] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.178563 29506 ts_tablet_manager.cc:1428] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:29.178735 29501 ts_tablet_manager.cc:1428] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:33:29.178737 29500 tablet_bootstrap.cc:492] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: No bootstrap required, opened a new log
I20260925 04:33:29.178812 29500 ts_tablet_manager.cc:1397] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:33:29.179246 29500 raft_consensus.cc:348] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.179344 29500 raft_consensus.cc:374] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.179376 29500 raft_consensus.cc:729] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f0859cd2a4a36851a659fe401f685, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.179504 29500 consensus_queue.cc:260] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.179786 29500 ts_tablet_manager.cc:1428] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260925 04:33:29.408725 29190 tablet.cc:2369] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:29.504015 29505 raft_consensus.cc:482] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:29.504206 29505 raft_consensus.cc:504] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.505438 29505 leader_election.cc:288] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505)
I20260925 04:33:29.509431 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.505864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36283 (local address 127.27.46.194:38505)
0925 04:33:29.505995 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.506001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.506285 (+ 284us) server_negotiation.cc:407] Connection header received
0925 04:33:29.506433 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.506438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.506484 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.506564 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:29.506813 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.507283 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508430 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.508819 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508950 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.509008 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.509228 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.509303 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:29.509431 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.505819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36657 (local address 127.27.46.193:34339)
0925 04:33:29.505975 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.505980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.506163 (+ 183us) server_negotiation.cc:407] Connection header received
0925 04:33:29.506329 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.506332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.506373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.506455 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:29.506733 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.507217 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508423 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.508807 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508930 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.508991 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.509200 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.509275 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:33:29.509478 29530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.505719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34339 (local address 127.27.46.195:36657)
0925 04:33:29.506066 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.506078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.506209 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.506476 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.506479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.506494 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:29.506623 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.506628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.507351 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.507358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.508295 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.508305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.508925 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.509058 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.509086 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.509179 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.509268 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":77,"thread_start_us":87,"threads_started":1}
I20260925 04:33:29.509485 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.505773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38505 (local address 127.27.46.195:36283)
0925 04:33:29.506188 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.506200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.506320 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.506578 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.506581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.506598 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:29.506705 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.506710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.507420 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.507425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.508302 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.508312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.508943 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.508946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.509102 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.509120 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.509208 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.509304 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":105,"thread_start_us":62,"threads_started":1}
I20260925 04:33:29.510020 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fdaf6e7e94fec9d2c11f36c7acf14" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" is_pre_election: true
I20260925 04:33:29.510176 29143 raft_consensus.cc:2454] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 0.
I20260925 04:33:29.510102 29287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fdaf6e7e94fec9d2c11f36c7acf14" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4187c02f0342a09bd79395cf400905" is_pre_election: true
I20260925 04:33:29.510406 29287 raft_consensus.cc:2454] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 0.
I20260925 04:33:29.510576 29361 leader_election.cc:302] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f0859cd2a4a36851a659fe401f685, 2ab545520a2b42e3a8fc74dcaf8a735a; no voters:
I20260925 04:33:29.510891 29505 raft_consensus.cc:2781] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:29.510946 29505 raft_consensus.cc:482] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:29.510972 29505 raft_consensus.cc:3037] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.512151 29505 raft_consensus.cc:504] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.512552 29505 leader_election.cc:288] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505)
I20260925 04:33:29.512881 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fdaf6e7e94fec9d2c11f36c7acf14" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
I20260925 04:33:29.512897 29287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043fdaf6e7e94fec9d2c11f36c7acf14" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4187c02f0342a09bd79395cf400905"
I20260925 04:33:29.512980 29143 raft_consensus.cc:3037] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.513010 29287 raft_consensus.cc:3037] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.513967 29143 raft_consensus.cc:2454] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 1.
I20260925 04:33:29.514057 29287 raft_consensus.cc:2454] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 1.
I20260925 04:33:29.514326 29361 leader_election.cc:302] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f0859cd2a4a36851a659fe401f685, 2ab545520a2b42e3a8fc74dcaf8a735a; no voters:
I20260925 04:33:29.514593 29505 raft_consensus.cc:2781] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:29.514760 29505 raft_consensus.cc:686] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 1 LEADER]: Becoming Leader. State: Replica: 045f0859cd2a4a36851a659fe401f685, State: Running, Role: LEADER
I20260925 04:33:29.515048 29505 consensus_queue.cc:237] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.517357 28982 catalog_manager.cc:5495] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045f0859cd2a4a36851a659fe401f685 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } health_report { overall_health: HEALTHY } } }
Sep 25 04:33:29 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:29.571329 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.563121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.27.46.254:44673)
0925 04:33:29.563382 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.563389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.563403 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:29.563521 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.563524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.563563 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.563646 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:29.564293 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.564837 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.565730 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.565951 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.566064 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.568249 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:29.568252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:29.568254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:29.568275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:29.569664 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.570085 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.570087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.570088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.570162 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.570430 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.570432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.570433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.570596 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:29.570667 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.570980 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.571169 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
I20260925 04:33:29.571329 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.563062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44673 (local address 127.0.0.1:55668)
0925 04:33:29.563344 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.563355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.563441 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.563705 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.563707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.564088 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:29.564179 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.564185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.564981 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.564985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.565614 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.565623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.566064 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.566069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.566152 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.566750 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:29.566762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:29.568127 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:29.569758 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:29.569766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:29.569772 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:29.569994 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:29.570246 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:29.570248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:29.570250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:29.570325 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:29.570684 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:29.570690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:29.570794 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.570934 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.571105 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260925 04:33:29.575387 28982 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:55668:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:33:29.576203 28982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:29.580943 29267 tablet_service.cc:1467] Processing CreateTablet for tablet 6f29642211eb4e51b9130dba15263973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29fb1db7779740b6a37ad630f13d80a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:29.580956 29411 tablet_service.cc:1467] Processing CreateTablet for tablet 6f29642211eb4e51b9130dba15263973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29fb1db7779740b6a37ad630f13d80a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:29.580935 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 6f29642211eb4e51b9130dba15263973 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29fb1db7779740b6a37ad630f13d80a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:29.581328 29122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f29642211eb4e51b9130dba15263973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.581331 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f29642211eb4e51b9130dba15263973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.581331 29267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f29642211eb4e51b9130dba15263973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:29.583688 29501 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: Bootstrap starting.
I20260925 04:33:29.584082 29500 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: Bootstrap starting.
I20260925 04:33:29.584489 29506 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: Bootstrap starting.
I20260925 04:33:29.584862 29501 tablet_bootstrap.cc:654] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.585317 29500 tablet_bootstrap.cc:654] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.585489 29506 tablet_bootstrap.cc:654] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:29.586733 29501 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: No bootstrap required, opened a new log
I20260925 04:33:29.586795 29506 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: No bootstrap required, opened a new log
I20260925 04:33:29.586833 29501 ts_tablet_manager.cc:1397] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:33:29.586861 29506 ts_tablet_manager.cc:1397] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 04:33:29.587244 29501 raft_consensus.cc:348] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.587244 29506 raft_consensus.cc:348] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.587366 29501 raft_consensus.cc:374] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.587366 29506 raft_consensus.cc:374] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.587401 29501 raft_consensus.cc:729] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab545520a2b42e3a8fc74dcaf8a735a, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.587401 29506 raft_consensus.cc:729] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4187c02f0342a09bd79395cf400905, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.587532 29501 consensus_queue.cc:260] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.587536 29506 consensus_queue.cc:260] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.587796 29506 ts_tablet_manager.cc:1428] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:29.587796 29501 ts_tablet_manager.cc:1428] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:33:29.588397 29504 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.588547 29504 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.588814 29518 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.588941 29518 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.590034 29500 tablet_bootstrap.cc:492] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: No bootstrap required, opened a new log
I20260925 04:33:29.590124 29500 ts_tablet_manager.cc:1397] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 04:33:29.590545 29500 raft_consensus.cc:348] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.590662 29500 raft_consensus.cc:374] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:29.590708 29500 raft_consensus.cc:729] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f0859cd2a4a36851a659fe401f685, State: Initialized, Role: FOLLOWER
I20260925 04:33:29.590832 29500 consensus_queue.cc:260] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.591117 29500 ts_tablet_manager.cc:1428] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:29.591106 29505 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.591265 29505 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.737833 29505 raft_consensus.cc:482] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:29.737963 29505 raft_consensus.cc:504] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.738426 29505 leader_election.cc:288] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505)
I20260925 04:33:29.738893 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f29642211eb4e51b9130dba15263973" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" is_pre_election: true
I20260925 04:33:29.738893 29287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f29642211eb4e51b9130dba15263973" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4187c02f0342a09bd79395cf400905" is_pre_election: true
I20260925 04:33:29.739087 29287 raft_consensus.cc:2454] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 0.
I20260925 04:33:29.739086 29143 raft_consensus.cc:2454] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 0.
I20260925 04:33:29.739379 29360 leader_election.cc:302] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f0859cd2a4a36851a659fe401f685, dd4187c02f0342a09bd79395cf400905; no voters:
I20260925 04:33:29.739583 29505 raft_consensus.cc:2781] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:29.739661 29505 raft_consensus.cc:482] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:29.739717 29505 raft_consensus.cc:3037] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.740842 29505 raft_consensus.cc:504] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.741214 29505 leader_election.cc:288] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339), dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505)
I20260925 04:33:29.741627 29287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f29642211eb4e51b9130dba15263973" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4187c02f0342a09bd79395cf400905"
I20260925 04:33:29.741621 29143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f29642211eb4e51b9130dba15263973" candidate_uuid: "045f0859cd2a4a36851a659fe401f685" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a"
I20260925 04:33:29.741751 29143 raft_consensus.cc:3037] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.741760 29287 raft_consensus.cc:3037] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:29.742698 29143 raft_consensus.cc:2454] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 1.
I20260925 04:33:29.742813 29287 raft_consensus.cc:2454] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045f0859cd2a4a36851a659fe401f685 in term 1.
I20260925 04:33:29.743007 29361 leader_election.cc:302] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f0859cd2a4a36851a659fe401f685, 2ab545520a2b42e3a8fc74dcaf8a735a; no voters:
I20260925 04:33:29.743165 29505 raft_consensus.cc:2781] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:29.743264 29505 raft_consensus.cc:686] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [term 1 LEADER]: Becoming Leader. State: Replica: 045f0859cd2a4a36851a659fe401f685, State: Running, Role: LEADER
I20260925 04:33:29.743403 29505 consensus_queue.cc:237] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } }
I20260925 04:33:29.744067 29531 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: TxnStatusTablet state changed. Reason: New leader 045f0859cd2a4a36851a659fe401f685. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.744174 29531 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:29.744525 29545 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:33:29.744699 29545 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:33:29.745201 28982 catalog_manager.cc:5495] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045f0859cd2a4a36851a659fe401f685 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } health_report { overall_health: HEALTHY } } }
I20260925 04:33:29.779834 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.775965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.27.46.195:43747)
0925 04:33:29.776236 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.776242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.776256 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:29.776326 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.776328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.776377 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.776497 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:29.776772 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.777302 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.778306 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.778559 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.778679 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.778970 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.779430 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.779551 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.779625 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.779676 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260925 04:33:29.779816 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.775842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:43747 (local address 127.0.0.1:54616)
0925 04:33:29.775978 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.775996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.776141 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.776506 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.776509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.776525 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:29.776645 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.776650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.777430 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.777435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.778139 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.778147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.778681 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.778685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.778785 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.778825 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.779532 (+ 707us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.779548 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.779615 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.779698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 04:33:29.782718 29143 raft_consensus.cc:1261] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Refusing update from remote peer 045f0859cd2a4a36851a659fe401f685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.782946 29287 raft_consensus.cc:1261] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Refusing update from remote peer 045f0859cd2a4a36851a659fe401f685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.783246 29531 consensus_queue.cc:1035] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.783476 29505 consensus_queue.cc:1035] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.784915 29504 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: TxnStatusTablet state changed. Reason: New leader 045f0859cd2a4a36851a659fe401f685. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.785041 29504 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.786461 29518 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: TxnStatusTablet state changed. Reason: New leader 045f0859cd2a4a36851a659fe401f685. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.786609 29518 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.787760 29505 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.787864 29505 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:29.789148 29550 mvcc.cc:205] Tried to move back new op lower bound from 7333113076864929792 to 7333113076710383616. Current Snapshot: MvccSnapshot[applied={T|T < 7333113076864929792}]
I20260925 04:33:29.790201 29518 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.790305 29518 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P dd4187c02f0342a09bd79395cf400905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.789655 29504 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.790344 29504 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 2ab545520a2b42e3a8fc74dcaf8a735a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:29.788353 29531 tablet_replica.cc:439] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "045f0859cd2a4a36851a659fe401f685" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 } } peers { permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 } } peers { permanent_uuid: "045f0859cd2a4a36851a659fe401f685" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 43747 } } }
I20260925 04:33:29.791036 29531 tablet_replica.cc:442] T 6f29642211eb4e51b9130dba15263973 P 045f0859cd2a4a36851a659fe401f685: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:29.798302 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.794800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.27.46.195:43747)
0925 04:33:29.794913 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.794917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.795053 (+ 136us) server_negotiation.cc:407] Connection header received
0925 04:33:29.795144 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.795147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.795193 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.795272 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:29.795480 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.796252 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.797032 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.797384 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.797486 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.797885 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.797964 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.798036 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.798125 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.798174 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:33:29.809005 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.805790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:59313 (local address 127.27.46.195:43747)
0925 04:33:29.805917 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.805921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.806092 (+ 171us) server_negotiation.cc:407] Connection header received
0925 04:33:29.806242 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.806247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.806293 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.806371 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:29.806619 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.807105 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.807969 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.808179 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.808287 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.808575 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.808641 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.808718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.808804 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.808841 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:29.809036 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.805604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:43747 (local address 127.27.46.195:59313)
0925 04:33:29.806007 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:29.806019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:29.806139 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:29.806379 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:29.806382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:29.806388 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:29.806505 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.806520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.807213 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.807217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.807844 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:29.807852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.808287 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.808290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:29.808392 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.808433 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:29.808721 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:29.808728 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:33:29.808788 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 04:33:29.808855 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":93,"threads_started":1}
I20260925 04:33:29.816601 29287 raft_consensus.cc:1261] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Refusing update from remote peer 045f0859cd2a4a36851a659fe401f685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.816642 29143 raft_consensus.cc:1261] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Refusing update from remote peer 045f0859cd2a4a36851a659fe401f685: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:29.817062 29558 consensus_queue.cc:1035] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4187c02f0342a09bd79395cf400905" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.817242 29531 consensus_queue.cc:1035] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab545520a2b42e3a8fc74dcaf8a735a" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:29.863307 28983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:29.863493 28983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:29.865870 28983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80131fa4772041d0ba1ce55fc987fd25: Sending DeleteTablet for 3 replicas of tablet 043fdaf6e7e94fec9d2c11f36c7acf14
I20260925 04:33:29.866439 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40264
I20260925 04:33:29.866791 29267 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:46172
I20260925 04:33:29.866824 29410 tablet_service.cc:1514] Processing DeleteTablet for tablet 043fdaf6e7e94fec9d2c11f36c7acf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:54606
I20260925 04:33:29.867163 29568 tablet_replica.cc:330] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: stopping tablet replica
I20260925 04:33:29.867184 29569 tablet_replica.cc:330] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: stopping tablet replica
I20260925 04:33:29.867309 29569 raft_consensus.cc:2229] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:29.867350 29568 raft_consensus.cc:2229] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:29.867600 29568 raft_consensus.cc:2258] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.867606 29569 raft_consensus.cc:2258] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.868299 29569 ts_tablet_manager.cc:1905] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.868712 29567 tablet_replica.cc:330] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: stopping tablet replica
I20260925 04:33:29.868872 29567 raft_consensus.cc:2229] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:29.869009 29567 raft_consensus.cc:2258] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:29.869092 29568 ts_tablet_manager.cc:1905] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.869704 29567 ts_tablet_manager.cc:1905] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:29.871361 29569 ts_tablet_manager.cc:1918] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:29.871444 29569 log.cc:1196] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/043fdaf6e7e94fec9d2c11f36c7acf14
I20260925 04:33:29.871748 29569 ts_tablet_manager.cc:1939] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 045f0859cd2a4a36851a659fe401f685: Deleting consensus metadata
I20260925 04:33:29.872437 28967 catalog_manager.cc:4841] TS 045f0859cd2a4a36851a659fe401f685 (127.27.46.195:43747): tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (table test-table [id=8682bf03403e4111b8ec478ad6a9697f]) successfully deleted
I20260925 04:33:29.872597 29568 ts_tablet_manager.cc:1918] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:29.872741 29568 log.cc:1196] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/043fdaf6e7e94fec9d2c11f36c7acf14
I20260925 04:33:29.872852 29567 ts_tablet_manager.cc:1918] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:29.872951 29567 log.cc:1196] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/043fdaf6e7e94fec9d2c11f36c7acf14
I20260925 04:33:29.873185 29568 ts_tablet_manager.cc:1939] T 043fdaf6e7e94fec9d2c11f36c7acf14 P dd4187c02f0342a09bd79395cf400905: Deleting consensus metadata
I20260925 04:33:29.873221 29567 ts_tablet_manager.cc:1939] T 043fdaf6e7e94fec9d2c11f36c7acf14 P 2ab545520a2b42e3a8fc74dcaf8a735a: Deleting consensus metadata
I20260925 04:33:29.873901 28967 catalog_manager.cc:4841] TS dd4187c02f0342a09bd79395cf400905 (127.27.46.194:38505): tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (table test-table [id=8682bf03403e4111b8ec478ad6a9697f]) successfully deleted
I20260925 04:33:29.873901 28968 catalog_manager.cc:4841] TS 2ab545520a2b42e3a8fc74dcaf8a735a (127.27.46.193:34339): tablet 043fdaf6e7e94fec9d2c11f36c7acf14 (table test-table [id=8682bf03403e4111b8ec478ad6a9697f]) successfully deleted
Sep 25 04:33:29 dist-test-slave-r8r3 krb5kdc[28938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:29.885777 29496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.872907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.27.46.193:34339)
0925 04:33:29.873032 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.873038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.873711 (+ 673us) server_negotiation.cc:407] Connection header received
0925 04:33:29.873933 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.873937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.873987 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.874038 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:29.875043 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.875558 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.876816 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.877068 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.877162 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.882117 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:29.882128 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:29.882142 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:29.882168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:29.883788 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.884251 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.884256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.884258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.884336 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.884663 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.884666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.884667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.884950 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:29.885187 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.885344 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.885470 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:33:29.886324 29174 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40266
I20260925 04:33:29.896831 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:29.888093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55676 (local address 127.27.46.254:44673)
0925 04:33:29.888214 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:29.888218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:29.888548 (+ 330us) server_negotiation.cc:407] Connection header received
0925 04:33:29.888645 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:29.888648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:29.888703 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:29.888814 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:29.889570 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.890454 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.891369 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:29.891718 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:29.891833 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:29.893058 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:29.893061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:29.893064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:29.893093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:29.895031 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.895488 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.895492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.895494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.895594 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:29.895913 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:29.895916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:29.895918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:29.896154 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:29.896299 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:29.896525 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 04:33:29.896664 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 04:33:29.897269 28983 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55676
I20260925 04:33:29.898774 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29056
I20260925 04:33:29.909106 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29199
I20260925 04:33:29.917426 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29343
I20260925 04:33:29.923862 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 28951
2026-09-25T04:33:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3144 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:29 dist-test-slave-r8r3 krb5kdc[29576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:29 dist-test-slave-r8r3 krb5kdc[29576](info): set up 2 sockets
Sep 25 04:33:29 dist-test-slave-r8r3 krb5kdc[29576](info): commencing operation
krb5kdc: starting...
W20260925 04:33:30.973320 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:31.094103 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34257
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:31.206880 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:31.207137 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:31.207206 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:31.211558 29589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:31.211624 29589 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:33:31.211642 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:31.211659 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:31.211692 29589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:31.211707 29589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:31.216908 29589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:34257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:31.218061 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:31.219110 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:31.227491 29595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.227491 29598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.227528 29596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:31.227711 29589 server_base.cc:1034] running on GCE node
I20260925 04:33:31.228374 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:31.229568 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:31.230744 29589 hybrid_clock.cc:648] HybridClock initialized: now 1790310811230731 us; error 22 us; skew 500 ppm
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:31.233299 29589 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:31.234181 29589 webserver.cc:460] Webserver started at http://127.27.46.254:36783/ using document root <none> and password file <none>
I20260925 04:33:31.234740 29589 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:31.234830 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:31.235062 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:31.237112 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "944613e8e32b4775bcb0582e223a98e6"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "2018cdc42fb3f4913977f4130f2fe827"
server_key_iv: "adde12ff7903c48f53a4239c9c2e85a5"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.237754 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "944613e8e32b4775bcb0582e223a98e6"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "2018cdc42fb3f4913977f4130f2fe827"
server_key_iv: "adde12ff7903c48f53a4239c9c2e85a5"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.241256 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:33:31.243481 29605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.244629 29589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:33:31.244776 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "944613e8e32b4775bcb0582e223a98e6"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "2018cdc42fb3f4913977f4130f2fe827"
server_key_iv: "adde12ff7903c48f53a4239c9c2e85a5"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.244897 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:31.256096 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:31.258556 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:31.258752 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:31.266233 29589 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:34257
I20260925 04:33:31.266254 29669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:34257 every 8 connection(s)
I20260925 04:33:31.267095 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:31.269716 29670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:31.269735 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29589
I20260925 04:33:31.269832 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:31.270083 27835 external_mini_cluster.cc:1428] Setting key 0a32e7ee0599debb135dde392505c20d
I20260925 04:33:31.275602 29670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Bootstrap starting.
I20260925 04:33:31.277902 29670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Neither blocks nor log segments found. Creating new log.
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:31.278811 29670 log.cc:824] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:31.280664 29670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: No bootstrap required, opened a new log
I20260925 04:33:31.283497 29670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } }
I20260925 04:33:31.283782 29670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:31.283886 29670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944613e8e32b4775bcb0582e223a98e6, State: Initialized, Role: FOLLOWER
I20260925 04:33:31.284315 29670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } }
I20260925 04:33:31.284483 29670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:31.284576 29670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:31.284696 29670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:31.284930 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.271435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.27.46.254:34257)
0925 04:33:31.271867 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.271880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.271913 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:33:31.272747 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.272766 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.273084 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.273399 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.274209 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.275124 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.276013 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.276459 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.276635 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.279416 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.279421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.279433 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.279459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.281764 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.282309 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.282321 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.282326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.282451 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.282738 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.282742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.282743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.283130 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.283340 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.283648 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.284006 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
I20260925 04:33:31.285693 29670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } }
I20260925 04:33:31.286098 29670 leader_election.cc:302] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944613e8e32b4775bcb0582e223a98e6; no voters:
I20260925 04:33:31.286443 29670 leader_election.cc:288] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:31.286541 29675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:31.286736 29675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [term 1 LEADER]: Becoming Leader. State: Replica: 944613e8e32b4775bcb0582e223a98e6, State: Running, Role: LEADER
I20260925 04:33:31.287092 29675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } }
I20260925 04:33:31.287633 29670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:31.288209 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944613e8e32b4775bcb0582e223a98e6. Latest consensus state: current_term: 1 leader_uuid: "944613e8e32b4775bcb0582e223a98e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } } }
I20260925 04:33:31.288316 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:31.288760 29676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "944613e8e32b4775bcb0582e223a98e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944613e8e32b4775bcb0582e223a98e6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34257 } } }
I20260925 04:33:31.288865 29676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:31.289881 29682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:31.292891 29682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:31.298482 29682 catalog_manager.cc:1347] Generated new cluster ID: b355b313a21e48948a59a0f8bfe3c082
I20260925 04:33:31.298554 29682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:31.318318 29682 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:31.319741 29682 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:31.327580 29682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Generated new TSK 0
I20260925 04:33:31.328436 29682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:31.354662 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:34257
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:31.465202 29697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:31.465440 29697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:31.465485 29697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:31.469376 29697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:31.469484 29697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:31.469601 29697 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:33:31.469653 29697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:31.469712 29697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:31.474655 29697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:31.475710 29697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:31.476708 29697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:31.486105 29706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.486083 29704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:31.486167 29697 server_base.cc:1034] running on GCE node
W20260925 04:33:31.486104 29703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:31.486724 29697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:31.487268 29697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:31.488456 29697 hybrid_clock.cc:648] HybridClock initialized: now 1790310811488436 us; error 35 us; skew 500 ppm
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:31.491231 29697 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:31.492007 29697 webserver.cc:460] Webserver started at http://127.27.46.193:46501/ using document root <none> and password file <none>
I20260925 04:33:31.492545 29697 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:31.492632 29697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:31.492914 29697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:31.494812 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "47a8e56f451847618fe5a21fa16872fd"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "192a09a20d9b483a47b6a0eaeed7d6a6"
server_key_iv: "14ac4ec992a98eadda2f1c85469758b9"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.495322 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "47a8e56f451847618fe5a21fa16872fd"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "192a09a20d9b483a47b6a0eaeed7d6a6"
server_key_iv: "14ac4ec992a98eadda2f1c85469758b9"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.498842 29697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:31.501078 29713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.502245 29697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:31.502380 29697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "47a8e56f451847618fe5a21fa16872fd"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "192a09a20d9b483a47b6a0eaeed7d6a6"
server_key_iv: "14ac4ec992a98eadda2f1c85469758b9"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.502486 29697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:31.513374 29697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:31.515956 29697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:31.516157 29697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:31.517556 29697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:31.517643 29697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.517711 29697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:31.517763 29697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:31.528882 29697 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:37299
I20260925 04:33:31.528981 29834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:37299 every 8 connection(s)
I20260925 04:33:31.530085 29697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:31.530547 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29697
I20260925 04:33:31.530661 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:31.530975 27835 external_mini_cluster.cc:1428] Setting key 3300238827b162106d9c8ac0c4fdfc8c
I20260925 04:33:31.532369 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.518430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:37525 (local address 127.27.46.254:34257)
0925 04:33:31.518556 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.518559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.519564 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 04:33:31.520591 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.520595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.520654 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.520713 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.522508 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.523417 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.524785 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.525223 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.525344 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.528195 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.528201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.528207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.528244 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.529959 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.530484 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.530490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.530495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.530596 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.531109 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.531113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.531114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.531335 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.531429 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.532091 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.532204 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:33:31.533234 29722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.518946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.193:37525)
0925 04:33:31.519406 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.519445 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.520327 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.521203 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.521215 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.521704 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.522306 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.522326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.523563 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.523567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.524651 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.524660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.525424 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.525429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.525728 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.526359 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.526376 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.528018 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.530077 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.530082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.530093 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.530381 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.530771 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.530774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.530776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.530928 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.531485 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.531492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.531819 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.532020 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.532376 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260925 04:33:31.534595 29835 heartbeater.cc:347] Connected to a master server at 127.27.46.254:34257
I20260925 04:33:31.534747 29835 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:31.535291 29835 heartbeater.cc:511] Master 127.27.46.254:34257 requested a full tablet report, sending...
I20260925 04:33:31.536952 29622 ts_manager.cc:194] Registered new tserver with Master: 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299)
I20260925 04:33:31.538623 29622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:37525
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:31.544706 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.536628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:50981 (local address 127.27.46.254:34257)
0925 04:33:31.536739 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.536743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.537752 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 04:33:31.537874 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.537877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.537920 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.537999 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.538417 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.538983 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.539965 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.540190 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.540292 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.541352 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.541356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.541358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.541387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.542998 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.543427 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.543432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.543434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.543511 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.543823 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.543828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.543830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.544025 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.544136 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.544443 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.544576 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:33:31.544781 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.536601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.193:50981)
0925 04:33:31.537667 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.537678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.537786 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.538015 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.538018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.538215 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.538305 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.538310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.539107 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.539110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.539838 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.539847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.540298 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.540302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.540400 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.540892 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.540913 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.541215 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.543126 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.543129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.543131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.543326 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.543627 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.543630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.543631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.543739 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.544126 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.544129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.544227 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.544345 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.544487 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:31.560462 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:34257
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:31.668408 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:31.668707 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:31.668790 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:31.672660 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:31.672755 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:31.672828 29844 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:33:31.672883 29844 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:31.672962 29844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:31.678141 29844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:31.679194 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:31.680176 29844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:31.689463 29852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.689538 29850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.689458 29849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:31.689739 29844 server_base.cc:1034] running on GCE node
I20260925 04:33:31.690284 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:31.690843 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:31.691995 29844 hybrid_clock.cc:648] HybridClock initialized: now 1790310811691966 us; error 41 us; skew 500 ppm
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:31.694759 29844 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:31.695536 29844 webserver.cc:460] Webserver started at http://127.27.46.194:39407/ using document root <none> and password file <none>
I20260925 04:33:31.696115 29844 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:31.696175 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:31.696331 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:31.698675 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "e6c2299f45d846f6b6e27c39b89defc7"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "c8771148ddde091d1467d7d9d7885dc0"
server_key_iv: "c17e9d2c221b6a9ad6115aba5d33d118"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.699216 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "e6c2299f45d846f6b6e27c39b89defc7"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "c8771148ddde091d1467d7d9d7885dc0"
server_key_iv: "c17e9d2c221b6a9ad6115aba5d33d118"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.702924 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:31.705240 29859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.706580 29844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:31.706724 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "e6c2299f45d846f6b6e27c39b89defc7"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "c8771148ddde091d1467d7d9d7885dc0"
server_key_iv: "c17e9d2c221b6a9ad6115aba5d33d118"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.706839 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:31.721561 29844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:31.724460 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:31.724676 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:31.726265 29844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:31.726336 29844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.726403 29844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:31.726476 29844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:31.737820 29844 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:42503
I20260925 04:33:31.737943 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:42503 every 8 connection(s)
I20260925 04:33:31.738780 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:31.740212 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.727274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60365 (local address 127.27.46.254:34257)
0925 04:33:31.727420 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.727425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.728268 (+ 843us) server_negotiation.cc:407] Connection header received
0925 04:33:31.729303 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.729307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.729369 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.729432 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.731100 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.731830 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.733093 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.733331 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.733454 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.736565 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.736569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.736571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.736606 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.738210 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.738720 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.738723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.738724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.738819 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.739190 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.739194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.739196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.739414 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.739510 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.739987 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.740089 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:31.740959 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.727660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.194:60365)
0925 04:33:31.728094 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.728143 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.729054 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.729833 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.729840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.730294 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.730922 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.730945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.731958 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.731961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.732954 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.732964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.733544 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.733549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.733765 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.734483 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.734508 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.736391 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.738323 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.738329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.738343 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.738599 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.738910 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.738914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.738916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.739088 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.739562 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.739568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.739783 (+ 215us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.739934 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.740169 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
I20260925 04:33:31.742282 29981 heartbeater.cc:347] Connected to a master server at 127.27.46.254:34257
I20260925 04:33:31.742440 29981 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:31.742938 29981 heartbeater.cc:511] Master 127.27.46.254:34257 requested a full tablet report, sending...
I20260925 04:33:31.743976 29623 ts_manager.cc:194] Registered new tserver with Master: e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503)
I20260925 04:33:31.744747 29623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:60365
I20260925 04:33:31.746583 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29844
I20260925 04:33:31.746695 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:31.746965 27835 external_mini_cluster.cc:1428] Setting key e25d3b62f7f423373e4dfdf3fda277ea
I20260925 04:33:31.752111 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.744487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:45903 (local address 127.27.46.254:34257)
0925 04:33:31.745086 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.745092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.745104 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:33:31.745140 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.745143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.745173 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.745257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.745779 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.746261 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.747241 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.747585 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.747701 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.748922 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.748929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.748934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.748966 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.750502 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.750978 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.750981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.750982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.751049 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.751329 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.751331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.751332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.751536 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.751609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.751923 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.752006 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260925 04:33:31.752295 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.744359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.194:45903)
0925 04:33:31.744655 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.744670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.744756 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.745277 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.745280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.745468 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.745670 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.745677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.746387 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.746392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.747126 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.747135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.747813 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.747816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.747885 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.748399 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.748413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.748750 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.750618 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.750623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.750625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.750878 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.751134 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.751138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.751140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.751241 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.751634 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.751640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.751757 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.751875 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.752038 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":11,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:31.776636 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:34257
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:42505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:31.885715 29990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:31.886044 29990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:31.886124 29990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:31.890390 29990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:31.890448 29990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:31.890488 29990 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:33:31.890502 29990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:31.890533 29990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:31.895808 29990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:34257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:31.896777 29990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:31.897811 29990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:31.907312 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.907300 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:31.907346 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:31.907647 29990 server_base.cc:1034] running on GCE node
I20260925 04:33:31.908145 29990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:31.908800 29990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:31.909970 29990 hybrid_clock.cc:648] HybridClock initialized: now 1790310811909949 us; error 31 us; skew 500 ppm
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:31.912730 29990 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:31.913647 29990 webserver.cc:460] Webserver started at http://127.27.46.195:40151/ using document root <none> and password file <none>
I20260925 04:33:31.914224 29990 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:31.914315 29990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:31.914554 29990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:31.916558 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "09212cf3637c4ce987b9c55ef6255616"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "9223deb259d69b457d550506aedb2289"
server_key_iv: "ffbd877b904df97a523489bcd94f27be"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.917124 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "09212cf3637c4ce987b9c55ef6255616"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "9223deb259d69b457d550506aedb2289"
server_key_iv: "ffbd877b904df97a523489bcd94f27be"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.920933 29990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:33:31.923384 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.924649 29990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:31.924782 29990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "09212cf3637c4ce987b9c55ef6255616"
format_stamp: "Formatted at 2026-09-25 04:33:31 on dist-test-slave-r8r3"
server_key: "9223deb259d69b457d550506aedb2289"
server_key_iv: "ffbd877b904df97a523489bcd94f27be"
server_key_version: "encryptionkey@0"
I20260925 04:33:31.924892 29990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:31.936002 29990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:31.938781 29990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:31.938946 29990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:31.940464 29990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:31.940517 29990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:31.940582 29990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:31.940613 29990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:31.950722 29990 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:36383
I20260925 04:33:31.950781 30128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:36383 every 8 connection(s)
I20260925 04:33:31.951774 29990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:31.952554 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 29990
I20260925 04:33:31.952661 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:31.952950 27835 external_mini_cluster.cc:1428] Setting key b809f49873fcb16f577f2f2c84f108a3
I20260925 04:33:31.955116 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.941213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:45115 (local address 127.27.46.254:34257)
0925 04:33:31.941410 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.941435 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.942247 (+ 812us) server_negotiation.cc:407] Connection header received
0925 04:33:31.943439 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.943443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.943509 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.943572 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.945353 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.946299 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.947559 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.947932 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.948058 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.950553 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.950556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.950558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.950590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.952112 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.953322 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.953326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.953328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.953428 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.953857 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.953860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.953862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.954094 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.954175 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.954785 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.954951 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:33:31.955838 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.941654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.195:45115)
0925 04:33:31.942088 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.942137 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.943139 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.944010 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.944031 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.944472 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.945172 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.945190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.946451 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.946453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.947433 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.947444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.948110 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.948113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.948247 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.948826 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.948842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.950375 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.952222 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.952228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.952241 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.952629 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.953546 (+ 917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.953551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.953553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.953740 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.954188 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.954195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.954483 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.954681 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.954992 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260925 04:33:31.956895 30129 heartbeater.cc:347] Connected to a master server at 127.27.46.254:34257
I20260925 04:33:31.957039 30129 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:31.957571 30129 heartbeater.cc:511] Master 127.27.46.254:34257 requested a full tablet report, sending...
I20260925 04:33:31.958546 29622 ts_manager.cc:194] Registered new tserver with Master: 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:31.959197 29622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:45115
I20260925 04:33:31.965404 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.959084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:44519 (local address 127.27.46.254:34257)
0925 04:33:31.959211 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.959215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.959489 (+ 274us) server_negotiation.cc:407] Connection header received
0925 04:33:31.959566 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.959569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.959612 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.959701 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.960158 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.960608 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.961477 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.961755 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.961831 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.962720 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.962722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.962724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.962745 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.964155 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.964494 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.964496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.964497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.964567 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.964785 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.964787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.964788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.964940 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.965010 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.965236 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.965308 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:33:31.965559 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.959018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.27.46.195:44519)
0925 04:33:31.959366 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:31.959385 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:31.959492 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:31.959707 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:31.959709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:31.959933 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:31.960051 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.960059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.960733 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.960736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.961382 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:31.961391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.961848 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.961850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:31.961914 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.962320 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:31.962330 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:31.962598 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:31.964244 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.964246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.964247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.964416 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.964642 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:31.964644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:31.964645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:31.964712 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:31.965014 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:31.965021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:31.965086 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 04:33:31.965188 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:33:31.965342 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":92,"threads_started":1}
I20260925 04:33:31.967356 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:33:31 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:31.992136 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:31.983635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.27.46.254:34257)
0925 04:33:31.983772 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:31.983776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:31.983787 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:33:31.983824 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:31.983826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:31.983863 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:31.983955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:31.984676 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.985210 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.986169 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:31.986393 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:31.986510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:31.989084 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:31.989088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:31.989090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:31.989116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:31.990669 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.991075 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.991078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.991079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.991153 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:31.991380 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:31.991382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:31.991383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:31.991563 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:31.991659 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:31.991858 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 04:33:31.991951 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:33:31.995137 29623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:31.998174 29623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:32.013429 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.007208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42503 (local address 127.0.0.1:51728)
0925 04:33:32.007874 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.007888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.007978 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.008750 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.008753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.008767 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.008913 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.008920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.010087 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.010094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.011694 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.011705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.012865 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.012868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.012978 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.013022 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.013155 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.013275 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":25,"thread_start_us":116,"threads_started":1}
I20260925 04:33:32.013441 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.007385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.27.46.194:42503)
0925 04:33:32.007820 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.007832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.008149 (+ 317us) server_negotiation.cc:407] Connection header received
0925 04:33:32.008235 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.008241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.008436 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.008609 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.009057 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.009963 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.011847 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.012741 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.012857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.013049 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.013181 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.013274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":113,"threads_started":1}
I20260925 04:33:32.014472 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.006482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.0.0.1:40372)
0925 04:33:32.006884 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.006929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.007049 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.008633 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.008636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.008654 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.008907 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.008916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.011487 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.011491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.012604 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.012616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.014118 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.014122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.014207 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.014230 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.014274 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.014336 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":95,"thread_start_us":127,"threads_started":1}
I20260925 04:33:32.015055 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.006494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:37299 (local address 127.0.0.1:58942)
0925 04:33:32.007459 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.007472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.007594 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.008612 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.008618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.008643 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.008906 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.008916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.012446 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.012453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.013587 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.013599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.014741 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.014746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.014838 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.014863 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.014916 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.014964 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":73,"thread_start_us":74,"threads_started":1}
I20260925 04:33:32.015306 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.006605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.27.46.195:36383)
0925 04:33:32.008247 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.008259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.008277 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:33:32.008341 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.008345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.008487 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.008797 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.010156 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.011363 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.012766 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.013982 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.014879 (+ 897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.015025 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.015081 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.015162 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":111,"threads_started":1}
I20260925 04:33:32.015789 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.007669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.27.46.193:37299)
0925 04:33:32.008054 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.008068 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.008089 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:33:32.008197 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.008203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.008434 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.008609 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.011509 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.012311 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.013701 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.014607 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.015386 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.015572 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.015614 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.015681 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":94,"threads_started":1}
I20260925 04:33:32.016755 29912 tablet_service.cc:1467] Processing CreateTablet for tablet cd59cd502cd542c5af14d44963c2b706 (DEFAULT_TABLE table=test-table [id=3053b9c030d448f28746db004ae525c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.017915 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd59cd502cd542c5af14d44963c2b706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.017774 30057 tablet_service.cc:1467] Processing CreateTablet for tablet cd59cd502cd542c5af14d44963c2b706 (DEFAULT_TABLE table=test-table [id=3053b9c030d448f28746db004ae525c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.018085 29763 tablet_service.cc:1467] Processing CreateTablet for tablet cd59cd502cd542c5af14d44963c2b706 (DEFAULT_TABLE table=test-table [id=3053b9c030d448f28746db004ae525c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.018764 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd59cd502cd542c5af14d44963c2b706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.018870 29763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd59cd502cd542c5af14d44963c2b706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.025308 30154 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Bootstrap starting.
I20260925 04:33:32.026556 30155 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Bootstrap starting.
I20260925 04:33:32.026889 30156 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Bootstrap starting.
I20260925 04:33:32.028082 30154 tablet_bootstrap.cc:654] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.028749 30155 tablet_bootstrap.cc:654] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.028859 30156 tablet_bootstrap.cc:654] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.029168 30154 log.cc:824] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:32.029630 30155 log.cc:824] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:32.029623 30156 log.cc:824] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:32.031322 30154 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: No bootstrap required, opened a new log
I20260925 04:33:32.031410 30156 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: No bootstrap required, opened a new log
I20260925 04:33:32.031504 30155 tablet_bootstrap.cc:492] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: No bootstrap required, opened a new log
I20260925 04:33:32.031519 30154 ts_tablet_manager.cc:1397] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:33:32.031610 30156 ts_tablet_manager.cc:1397] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:33:32.031673 30155 ts_tablet_manager.cc:1397] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:33:32.035125 30156 raft_consensus.cc:348] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.035405 30156 raft_consensus.cc:374] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.035502 30156 raft_consensus.cc:729] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.035496 30155 raft_consensus.cc:348] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.035449 30154 raft_consensus.cc:348] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.035733 30155 raft_consensus.cc:374] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.035733 30154 raft_consensus.cc:374] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.035804 30154 raft_consensus.cc:729] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.035846 30155 raft_consensus.cc:729] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.036082 30156 consensus_queue.cc:260] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.036233 30155 consensus_queue.cc:260] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.036216 30154 consensus_queue.cc:260] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.036736 30129 heartbeater.cc:503] Master 127.27.46.254:34257 was elected leader, sending a full tablet report...
I20260925 04:33:32.037145 30155 ts_tablet_manager.cc:1428] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:33:32.037145 30154 ts_tablet_manager.cc:1428] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:33:32.037549 29835 heartbeater.cc:503] Master 127.27.46.254:34257 was elected leader, sending a full tablet report...
I20260925 04:33:32.038187 30156 ts_tablet_manager.cc:1428] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:33:32.038527 29981 heartbeater.cc:503] Master 127.27.46.254:34257 was elected leader, sending a full tablet report...
I20260925 04:33:32.159751 30162 raft_consensus.cc:482] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:32.160065 30162 raft_consensus.cc:504] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.161263 30162 leader_election.cc:288] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383), 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299)
I20260925 04:33:32.165066 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.161733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:50153 (local address 127.27.46.193:37299)
0925 04:33:32.161876 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.161882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.161899 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:33:32.161966 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.161968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.162011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.162105 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.162435 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.162948 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.164044 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.164455 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.164600 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.164665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.164856 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.164935 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:32.165066 29869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.161559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:37299 (local address 127.27.46.194:50153)
0925 04:33:32.161679 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.161697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.161795 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.162120 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.162123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.162172 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.162291 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.162299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.163074 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.163079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.163900 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.163910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.164588 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.164592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.164719 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.164740 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.164838 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.164926 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":41600}
I20260925 04:33:32.165566 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.161658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:56227 (local address 127.27.46.195:36383)
0925 04:33:32.161787 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.161793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.162116 (+ 323us) server_negotiation.cc:407] Connection header received
0925 04:33:32.162292 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.162296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.162347 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.162427 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.163039 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.163559 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.164568 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.164960 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.165091 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.165138 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.165337 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.165407 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:32.165904 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.161564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.27.46.194:56227)
0925 04:33:32.162021 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.162031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.162180 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.162440 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.162442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.162457 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.162915 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.162922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.163690 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.163696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.164436 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.164445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.165091 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.165095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.165197 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.165224 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.165725 (+ 501us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.165798 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":105,"spinlock_wait_cycles":5376,"thread_start_us":175,"threads_started":1}
I20260925 04:33:32.165858 29785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd59cd502cd542c5af14d44963c2b706" candidate_uuid: "e6c2299f45d846f6b6e27c39b89defc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a8e56f451847618fe5a21fa16872fd" is_pre_election: true
I20260925 04:33:32.166180 29785 raft_consensus.cc:2454] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c2299f45d846f6b6e27c39b89defc7 in term 0.
I20260925 04:33:32.166543 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd59cd502cd542c5af14d44963c2b706" candidate_uuid: "e6c2299f45d846f6b6e27c39b89defc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616" is_pre_election: true
I20260925 04:33:32.166793 29863 leader_election.cc:302] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:32.166873 30078 raft_consensus.cc:2454] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6c2299f45d846f6b6e27c39b89defc7 in term 0.
I20260925 04:33:32.167066 30162 raft_consensus.cc:2781] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:32.167172 30162 raft_consensus.cc:482] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:32.167230 30162 raft_consensus.cc:3037] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.168231 30162 raft_consensus.cc:504] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.168532 30162 leader_election.cc:288] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [CANDIDATE]: Term 1 election: Requested vote from peers 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383), 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299)
I20260925 04:33:32.168924 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd59cd502cd542c5af14d44963c2b706" candidate_uuid: "e6c2299f45d846f6b6e27c39b89defc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616"
I20260925 04:33:32.169003 29785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd59cd502cd542c5af14d44963c2b706" candidate_uuid: "e6c2299f45d846f6b6e27c39b89defc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a8e56f451847618fe5a21fa16872fd"
I20260925 04:33:32.169082 30078 raft_consensus.cc:3037] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.169153 29785 raft_consensus.cc:3037] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.170167 29785 raft_consensus.cc:2454] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c2299f45d846f6b6e27c39b89defc7 in term 1.
I20260925 04:33:32.170280 30078 raft_consensus.cc:2454] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6c2299f45d846f6b6e27c39b89defc7 in term 1.
I20260925 04:33:32.170490 29863 leader_election.cc:302] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:32.170670 30162 raft_consensus.cc:2781] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:32.170883 30162 raft_consensus.cc:686] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 LEADER]: Becoming Leader. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Running, Role: LEADER
I20260925 04:33:32.171192 30162 consensus_queue.cc:237] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } }
I20260925 04:33:32.174211 29622 catalog_manager.cc:5495] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194). New cstate: current_term: 1 leader_uuid: "e6c2299f45d846f6b6e27c39b89defc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } health_report { overall_health: HEALTHY } } }
W20260925 04:33:32.203318 30130 tablet.cc:2369] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:33:32 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:32.231492 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.223176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:34257 (local address 127.0.0.1:54328)
0925 04:33:32.223476 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.223488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.223643 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.223897 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.223900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.224317 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:32.224402 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.224408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.225168 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.225174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.225730 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.225739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.226171 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.226174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.226252 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.226811 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:32.226823 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:32.228264 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:32.230062 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:32.230071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:32.230076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:32.230303 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:32.230554 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:32.230556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:32.230557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:32.230633 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:32.230997 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:32.231002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:32.231113 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.231203 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.231310 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
I20260925 04:33:32.231492 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.223215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.27.46.254:34257)
0925 04:33:32.223380 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.223385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.223571 (+ 186us) server_negotiation.cc:407] Connection header received
0925 04:33:32.223742 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.223745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.223791 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.223874 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:32.224525 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.225062 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.225847 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.226062 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.226167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.228393 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:32.228396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:32.228398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:32.228420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:32.229953 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:32.230388 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:32.230391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:32.230392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:32.230462 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:32.230705 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:32.230707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:32.230708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:32.230896 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:32.230984 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.231268 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.231366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:32.235504 29622 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:54328:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:33:32.236227 29622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 04:33:32.239977 29982 tablet.cc:2369] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:32.241262 29912 tablet_service.cc:1467] Processing CreateTablet for tablet c4da77a40b8d48a2a39b7e44cb7f972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b93c81c4343c4b188e40ca64ce61868d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:32.241266 30057 tablet_service.cc:1467] Processing CreateTablet for tablet c4da77a40b8d48a2a39b7e44cb7f972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b93c81c4343c4b188e40ca64ce61868d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:32.241185 29763 tablet_service.cc:1467] Processing CreateTablet for tablet c4da77a40b8d48a2a39b7e44cb7f972d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b93c81c4343c4b188e40ca64ce61868d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:32.241979 29763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4da77a40b8d48a2a39b7e44cb7f972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.241972 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4da77a40b8d48a2a39b7e44cb7f972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.241984 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4da77a40b8d48a2a39b7e44cb7f972d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.244437 30155 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: Bootstrap starting.
I20260925 04:33:32.244843 30156 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: Bootstrap starting.
I20260925 04:33:32.244843 30154 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: Bootstrap starting.
I20260925 04:33:32.245554 30155 tablet_bootstrap.cc:654] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.245774 30154 tablet_bootstrap.cc:654] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.245772 30156 tablet_bootstrap.cc:654] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.246694 30155 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: No bootstrap required, opened a new log
I20260925 04:33:32.246762 30155 ts_tablet_manager.cc:1397] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:33:32.247315 30154 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: No bootstrap required, opened a new log
I20260925 04:33:32.247305 30155 raft_consensus.cc:348] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.247403 30154 ts_tablet_manager.cc:1397] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:32.247442 30155 raft_consensus.cc:374] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.247483 30155 raft_consensus.cc:729] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.247635 30155 consensus_queue.cc:260] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.247932 30155 ts_tablet_manager.cc:1428] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:32.247888 30162 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.247905 30154 raft_consensus.cc:348] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.248009 30162 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.248008 30154 raft_consensus.cc:374] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.248042 30154 raft_consensus.cc:729] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.248155 30154 consensus_queue.cc:260] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.248414 30161 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.248544 30161 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.248984 30154 ts_tablet_manager.cc:1428] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:32.249269 30156 tablet_bootstrap.cc:492] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: No bootstrap required, opened a new log
I20260925 04:33:32.249326 30156 ts_tablet_manager.cc:1397] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 04:33:32.249716 30156 raft_consensus.cc:348] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.249797 30156 raft_consensus.cc:374] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.249816 30156 raft_consensus.cc:729] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.249938 30156 consensus_queue.cc:260] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.250207 30156 ts_tablet_manager.cc:1428] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:32.250268 30160 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.250411 30160 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 04:33:32.283900 29836 tablet.cc:2369] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:32.389447 30160 raft_consensus.cc:482] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:32.389644 30160 raft_consensus.cc:504] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.390810 30160 leader_election.cc:288] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299), e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503)
I20260925 04:33:32.394933 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.391120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:37299 (local address 127.27.46.195:38673)
0925 04:33:32.391272 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.391290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.391401 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.391870 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.391872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.391914 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.392037 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.392042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.392772 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.392776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.393631 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.393640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.394447 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.394451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.394590 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.394619 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.394700 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.394786 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"spinlock_wait_cycles":5504}
I20260925 04:33:32.395161 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.391216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:38673 (local address 127.27.46.193:37299)
0925 04:33:32.391361 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.391366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.391382 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:33:32.391704 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.391710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.391766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.391852 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.392146 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.392638 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.393749 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.394322 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.394451 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.394504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.394932 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.395018 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:33:32.395460 30177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.391418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42503 (local address 127.27.46.195:52199)
0925 04:33:32.391756 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.391768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.391925 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.392188 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.392191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.392206 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.392330 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.392335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.393079 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.393085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.393936 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.393946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.395010 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.395015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.395126 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.395153 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.395238 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.395309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":41600,"thread_start_us":132,"threads_started":1}
I20260925 04:33:32.395448 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.391567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52199 (local address 127.27.46.194:42503)
0925 04:33:32.391692 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.391696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.391857 (+ 161us) server_negotiation.cc:407] Connection header received
0925 04:33:32.392045 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.392048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.392097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.392177 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.392447 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.392956 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.394098 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.394671 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.394787 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.394864 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.395268 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.395345 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:33:32.395566 29785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4da77a40b8d48a2a39b7e44cb7f972d" candidate_uuid: "09212cf3637c4ce987b9c55ef6255616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a8e56f451847618fe5a21fa16872fd" is_pre_election: true
I20260925 04:33:32.395737 29785 raft_consensus.cc:2454] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09212cf3637c4ce987b9c55ef6255616 in term 0.
I20260925 04:33:32.396075 30008 leader_election.cc:302] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09212cf3637c4ce987b9c55ef6255616, 47a8e56f451847618fe5a21fa16872fd; no voters:
I20260925 04:33:32.396080 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4da77a40b8d48a2a39b7e44cb7f972d" candidate_uuid: "09212cf3637c4ce987b9c55ef6255616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7" is_pre_election: true
I20260925 04:33:32.396308 29934 raft_consensus.cc:2454] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09212cf3637c4ce987b9c55ef6255616 in term 0.
I20260925 04:33:32.396370 30160 raft_consensus.cc:2781] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:32.396425 30160 raft_consensus.cc:482] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:32.396464 30160 raft_consensus.cc:3037] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.397627 30160 raft_consensus.cc:504] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.397924 30160 leader_election.cc:288] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [CANDIDATE]: Term 1 election: Requested vote from peers 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299), e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503)
I20260925 04:33:32.398311 29785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4da77a40b8d48a2a39b7e44cb7f972d" candidate_uuid: "09212cf3637c4ce987b9c55ef6255616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a8e56f451847618fe5a21fa16872fd"
I20260925 04:33:32.398335 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4da77a40b8d48a2a39b7e44cb7f972d" candidate_uuid: "09212cf3637c4ce987b9c55ef6255616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7"
I20260925 04:33:32.398437 29934 raft_consensus.cc:3037] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.398437 29785 raft_consensus.cc:3037] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.399564 29785 raft_consensus.cc:2454] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09212cf3637c4ce987b9c55ef6255616 in term 1.
I20260925 04:33:32.399566 29934 raft_consensus.cc:2454] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09212cf3637c4ce987b9c55ef6255616 in term 1.
I20260925 04:33:32.399829 30008 leader_election.cc:302] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09212cf3637c4ce987b9c55ef6255616, 47a8e56f451847618fe5a21fa16872fd; no voters:
I20260925 04:33:32.399991 30160 raft_consensus.cc:2781] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:32.400274 30160 raft_consensus.cc:686] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [term 1 LEADER]: Becoming Leader. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Running, Role: LEADER
I20260925 04:33:32.400616 30160 consensus_queue.cc:237] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.402786 30178 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: TxnStatusTablet state changed. Reason: New leader 09212cf3637c4ce987b9c55ef6255616. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.402900 30178 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:32.403190 30182 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:33:32.403390 30182 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:33:32.404116 29622 catalog_manager.cc:5495] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } health_report { overall_health: HEALTHY } } }
I20260925 04:33:32.441457 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.437655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.27.46.195:36383)
0925 04:33:32.437788 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.437791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.437805 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:32.438000 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.438002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.438063 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.438145 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.438416 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.438900 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.440068 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.440259 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.440330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.440648 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.441050 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.441131 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.441249 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.441293 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:33:32.441469 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.437488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.0.0.1:40384)
0925 04:33:32.437733 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.437749 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.437856 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.438184 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.438187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.438195 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:32.438314 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.438320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.439003 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.439006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.439955 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.439966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.440360 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.440363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.440461 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.440514 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.441140 (+ 626us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.441158 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.441229 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.441313 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 04:33:32.446782 29934 raft_consensus.cc:1261] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Refusing update from remote peer 09212cf3637c4ce987b9c55ef6255616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.446784 29785 raft_consensus.cc:1261] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Refusing update from remote peer 09212cf3637c4ce987b9c55ef6255616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.447443 30160 consensus_queue.cc:1035] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.447904 30178 consensus_queue.cc:1035] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.451876 30161 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: TxnStatusTablet state changed. Reason: New leader 09212cf3637c4ce987b9c55ef6255616. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.452008 30161 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.452525 30162 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: TxnStatusTablet state changed. Reason: New leader 09212cf3637c4ce987b9c55ef6255616. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.452663 30162 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.455081 30178 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.455185 30178 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:32.456136 30162 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.456243 30162 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P e6c2299f45d846f6b6e27c39b89defc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.460716 30178 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.460826 30178 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 09212cf3637c4ce987b9c55ef6255616: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:32.462584 30186 mvcc.cc:205] Tried to move back new op lower bound from 7333113087774994432 to 7333113087595331584. Current Snapshot: MvccSnapshot[applied={T|T < 7333113087774994432}]
I20260925 04:33:32.462805 30161 tablet_replica.cc:439] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09212cf3637c4ce987b9c55ef6255616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } } }
I20260925 04:33:32.462908 30161 tablet_replica.cc:442] T c4da77a40b8d48a2a39b7e44cb7f972d P 47a8e56f451847618fe5a21fa16872fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:32.467049 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.463043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.27.46.194:42503)
0925 04:33:32.463147 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.463152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.463165 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:33:32.463366 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.463369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.463433 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.463551 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.463839 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.464491 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.465382 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.465714 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.465827 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.466140 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.466651 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.466761 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.466863 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.466916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:33:32.477427 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.474033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:35707 (local address 127.27.46.195:36383)
0925 04:33:32.474144 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.474147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.474413 (+ 266us) server_negotiation.cc:407] Connection header received
0925 04:33:32.474543 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.474545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.474585 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.474645 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.474911 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.475341 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.476441 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.476639 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.476710 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.477063 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.477123 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.477168 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.477287 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.477327 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:33:32.477526 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.473954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.27.46.194:35707)
0925 04:33:32.474332 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.474346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.474451 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.474678 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.474682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.474689 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:32.474819 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.474824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.475435 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.475438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.476329 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.476340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.476745 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.476748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.476857 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.476907 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.477200 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.477206 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.477272 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.477349 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260925 04:33:32.484771 29785 raft_consensus.cc:1261] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Refusing update from remote peer e6c2299f45d846f6b6e27c39b89defc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.485008 30078 raft_consensus.cc:1261] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Refusing update from remote peer e6c2299f45d846f6b6e27c39b89defc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.485297 30164 consensus_queue.cc:1035] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.485528 30162 consensus_queue.cc:1035] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.507596 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.503700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42503 (local address 127.27.46.195:54389)
0925 04:33:32.503999 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.504016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.504112 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.504405 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.504407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.504414 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:32.504497 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.504502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.505518 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.505521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.506356 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.506367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.506964 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.506966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.507033 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.507078 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.507357 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.507360 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.507400 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.507459 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
I20260925 04:33:32.507609 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.503976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:54389 (local address 127.27.46.194:42503)
0925 04:33:32.504107 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.504112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.504125 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:33:32.504242 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.504246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.504295 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.504374 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.504610 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.505382 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.506521 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.506865 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.506966 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.507201 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.507281 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.507359 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.507413 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.507465 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:32.534669 29622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:32.534857 29622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:32.537619 29622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Sending DeleteTablet for 3 replicas of tablet cd59cd502cd542c5af14d44963c2b706
I20260925 04:33:32.538713 29763 tablet_service.cc:1514] Processing DeleteTablet for tablet cd59cd502cd542c5af14d44963c2b706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:58942
I20260925 04:33:32.539137 29912 tablet_service.cc:1514] Processing DeleteTablet for tablet cd59cd502cd542c5af14d44963c2b706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260925 04:33:32.539670 30057 tablet_service.cc:1514] Processing DeleteTablet for tablet cd59cd502cd542c5af14d44963c2b706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 04:33:32.540863 29622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:32.541293 29622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:32.546346 30057 tablet_service.cc:1467] Processing CreateTablet for tablet acbf7b6c0b95402980172048cb101654 (DEFAULT_TABLE table=test-table [id=17bd84d9f33244a9b142003eebbd8e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.546680 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbf7b6c0b95402980172048cb101654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.547178 29912 tablet_service.cc:1467] Processing CreateTablet for tablet acbf7b6c0b95402980172048cb101654 (DEFAULT_TABLE table=test-table [id=17bd84d9f33244a9b142003eebbd8e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.547425 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbf7b6c0b95402980172048cb101654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.549345 30156 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Bootstrap starting.
I20260925 04:33:32.549526 29763 tablet_service.cc:1467] Processing CreateTablet for tablet acbf7b6c0b95402980172048cb101654 (DEFAULT_TABLE table=test-table [id=17bd84d9f33244a9b142003eebbd8e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:32.549798 29763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acbf7b6c0b95402980172048cb101654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:32.549958 30155 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Bootstrap starting.
I20260925 04:33:32.550376 30156 tablet_bootstrap.cc:654] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.551054 30155 tablet_bootstrap.cc:654] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.552049 30156 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: No bootstrap required, opened a new log
I20260925 04:33:32.552073 30154 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Bootstrap starting.
I20260925 04:33:32.552132 30156 ts_tablet_manager.cc:1397] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:32.552573 30156 raft_consensus.cc:348] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.552687 30156 raft_consensus.cc:374] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.552721 30156 raft_consensus.cc:729] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.552878 30154 tablet_bootstrap.cc:654] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:32.552853 30156 consensus_queue.cc:260] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.553140 30156 ts_tablet_manager.cc:1428] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:32.553941 30206 tablet_replica.cc:330] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: stopping tablet replica
I20260925 04:33:32.554646 30155 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: No bootstrap required, opened a new log
I20260925 04:33:32.554735 30155 ts_tablet_manager.cc:1397] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 04:33:32.555227 30209 tablet_replica.cc:330] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: stopping tablet replica
I20260925 04:33:32.555248 30155 raft_consensus.cc:348] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.555374 30155 raft_consensus.cc:374] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.555418 30155 raft_consensus.cc:729] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.555552 30155 consensus_queue.cc:260] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.555603 30209 raft_consensus.cc:2229] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:32.555773 30209 raft_consensus.cc:2258] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:32.554917 30154 tablet_bootstrap.cc:492] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: No bootstrap required, opened a new log
I20260925 04:33:32.556226 30154 ts_tablet_manager.cc:1397] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:33:32.556335 30155 ts_tablet_manager.cc:1428] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:33:32.556461 30207 tablet_replica.cc:330] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: stopping tablet replica
I20260925 04:33:32.556703 30154 raft_consensus.cc:348] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.556824 30154 raft_consensus.cc:374] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:32.556867 30154 raft_consensus.cc:729] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Initialized, Role: FOLLOWER
I20260925 04:33:32.556885 30207 raft_consensus.cc:2229] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:32.556998 30154 consensus_queue.cc:260] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.557260 30154 ts_tablet_manager.cc:1428] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:32.557279 30207 raft_consensus.cc:2258] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:32.558698 30207 ts_tablet_manager.cc:1905] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:32.561597 30207 ts_tablet_manager.cc:1918] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:32.561743 30207 log.cc:1196] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/cd59cd502cd542c5af14d44963c2b706
I20260925 04:33:32.562122 30207 ts_tablet_manager.cc:1939] T cd59cd502cd542c5af14d44963c2b706 P e6c2299f45d846f6b6e27c39b89defc7: Deleting consensus metadata
I20260925 04:33:32.563365 29608 catalog_manager.cc:4841] TS e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503): tablet cd59cd502cd542c5af14d44963c2b706 (table test-table [id=3053b9c030d448f28746db004ae525c5]) successfully deleted
I20260925 04:33:32.564234 30206 raft_consensus.cc:2229] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:32.564455 30206 raft_consensus.cc:2258] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:32.565797 30206 ts_tablet_manager.cc:1905] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:32.567453 30209 ts_tablet_manager.cc:1905] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:32.570863 30209 ts_tablet_manager.cc:1918] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:32.570962 30209 log.cc:1196] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/cd59cd502cd542c5af14d44963c2b706
I20260925 04:33:32.571286 30209 ts_tablet_manager.cc:1939] T cd59cd502cd542c5af14d44963c2b706 P 09212cf3637c4ce987b9c55ef6255616: Deleting consensus metadata
I20260925 04:33:32.572422 30206 ts_tablet_manager.cc:1918] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:32.572448 29606 catalog_manager.cc:4841] TS 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383): tablet cd59cd502cd542c5af14d44963c2b706 (table test-table [id=3053b9c030d448f28746db004ae525c5]) successfully deleted
I20260925 04:33:32.572607 30206 log.cc:1196] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/cd59cd502cd542c5af14d44963c2b706
I20260925 04:33:32.572964 30206 ts_tablet_manager.cc:1939] T cd59cd502cd542c5af14d44963c2b706 P 47a8e56f451847618fe5a21fa16872fd: Deleting consensus metadata
I20260925 04:33:32.574119 29608 catalog_manager.cc:4841] TS 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299): tablet cd59cd502cd542c5af14d44963c2b706 (table test-table [id=3053b9c030d448f28746db004ae525c5]) successfully deleted
I20260925 04:33:32.621757 30161 raft_consensus.cc:482] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:32.621922 30161 raft_consensus.cc:504] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.623225 30161 leader_election.cc:288] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:32.628450 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.623453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42503 (local address 127.27.46.193:43409)
0925 04:33:32.624153 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.624164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.624276 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.624652 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.624654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.624670 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.624763 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.624767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.625472 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.625478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.627078 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.627090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.627996 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.628000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.628106 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.628134 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.628212 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.628280 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":482,"thread_start_us":105,"threads_started":1}
I20260925 04:33:32.628505 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.623628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:43409 (local address 127.27.46.194:42503)
0925 04:33:32.623755 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.623760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.624251 (+ 491us) server_negotiation.cc:407] Connection header received
0925 04:33:32.624404 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.624407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.624460 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.624507 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.624891 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.625352 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.627223 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.627849 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.627986 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.628042 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.628285 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.628364 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:32.629190 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.623686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:33085 (local address 127.27.46.195:36383)
0925 04:33:32.623812 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.623817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.623949 (+ 132us) server_negotiation.cc:407] Connection header received
0925 04:33:32.624149 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.624152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.624221 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.624294 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:32.624623 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.625389 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.627897 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.628510 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.628635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.628676 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.629008 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.629099 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:32.629415 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbf7b6c0b95402980172048cb101654" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7" is_pre_election: true
I20260925 04:33:32.629608 30213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.623453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.27.46.193:33085)
0925 04:33:32.623839 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.623856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.623999 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.624326 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.624329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.624347 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:32.624489 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.624498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.625551 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.625554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.627078 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.627090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.628726 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.628729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.628832 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.628852 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.629374 (+ 522us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.629454 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":137,"thread_start_us":120,"threads_started":1}
I20260925 04:33:32.629612 29934 raft_consensus.cc:2454] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:32.630002 29716 leader_election.cc:302] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:32.630086 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbf7b6c0b95402980172048cb101654" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616" is_pre_election: true
I20260925 04:33:32.630214 30078 raft_consensus.cc:2454] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:32.630292 30161 raft_consensus.cc:2781] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:32.630357 30161 raft_consensus.cc:482] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:32.630393 30161 raft_consensus.cc:3037] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.631678 30161 raft_consensus.cc:504] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.632047 30161 leader_election.cc:288] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Requested vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:32.632385 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbf7b6c0b95402980172048cb101654" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616"
I20260925 04:33:32.632422 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acbf7b6c0b95402980172048cb101654" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7"
I20260925 04:33:32.632475 30078 raft_consensus.cc:3037] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.632540 29934 raft_consensus.cc:3037] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:32.633335 30078 raft_consensus.cc:2454] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:32.633644 29934 raft_consensus.cc:2454] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:32.634181 29714 leader_election.cc:302] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09212cf3637c4ce987b9c55ef6255616, 47a8e56f451847618fe5a21fa16872fd; no voters:
I20260925 04:33:32.634392 30161 raft_consensus.cc:2781] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:32.634665 30161 raft_consensus.cc:686] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 1 LEADER]: Becoming Leader. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Running, Role: LEADER
I20260925 04:33:32.634918 30161 consensus_queue.cc:237] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:32.637470 29622 catalog_manager.cc:5495] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 47a8e56f451847618fe5a21fa16872fd (127.27.46.193). New cstate: current_term: 1 leader_uuid: "47a8e56f451847618fe5a21fa16872fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:32.668311 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.664605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.27.46.193:37299)
0925 04:33:32.664722 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.664726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.664740 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:32.664868 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.664871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.664934 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.665042 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.665287 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.665844 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.666813 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.667052 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.667164 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.667458 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.667900 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.668030 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.668107 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.668162 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:33:32.679739 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.676034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:36383 (local address 127.27.46.193:48903)
0925 04:33:32.676349 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.676362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.676449 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.676744 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.676745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.676751 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:32.676851 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.676856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.677866 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.677869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.678464 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.678472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.679073 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.679075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.679143 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.679184 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.679491 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.679495 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.679541 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.679602 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":110,"threads_started":1}
I20260925 04:33:32.679773 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.676165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48903 (local address 127.27.46.195:36383)
0925 04:33:32.676322 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.676326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.676435 (+ 109us) server_negotiation.cc:407] Connection header received
0925 04:33:32.676545 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.676548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.676613 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.676722 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.676962 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.677758 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.678588 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.678942 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.679051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.679318 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.679402 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.679482 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.679554 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.679617 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:33:32.686244 30078 raft_consensus.cc:1261] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.686250 29934 raft_consensus.cc:1261] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:32.686725 30161 consensus_queue.cc:1035] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.686944 30215 consensus_queue.cc:1035] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:32.691013 30185 mvcc.cc:205] Tried to move back new op lower bound from 7333113088758284288 to 7333113088555864064. Current Snapshot: MvccSnapshot[applied={T|T < 7333113088758284288}]
I20260925 04:33:32.707206 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.703285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:47439 (local address 127.27.46.193:37299)
0925 04:33:32.703386 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:32.703390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:32.703403 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:33:32.703443 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:32.703446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:32.703500 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:32.703583 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:32.704069 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.704673 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.706088 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.706412 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.706525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.706775 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.706853 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.706928 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:32.707063 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 04:33:32.707109 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:33:32.707334 30203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:32.703086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:37299 (local address 127.27.46.195:47439)
0925 04:33:32.703207 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:32.703223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:32.703306 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:32.703733 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:32.703736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:32.703744 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:32.703847 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.703854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.704787 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.704790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.705841 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:32.705855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:32.706524 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:32.706528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:32.706621 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:32.706650 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:32.706959 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:32.706969 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 04:33:32.707052 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:33:32.707161 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 04:33:32.708282 30186 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:33:34.273077 29619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:34.273238 29619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:34.275583 29619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Sending DeleteTablet for 3 replicas of tablet acbf7b6c0b95402980172048cb101654
I20260925 04:33:34.276120 29763 tablet_service.cc:1514] Processing DeleteTablet for tablet acbf7b6c0b95402980172048cb101654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:58942
I20260925 04:33:34.276165 29912 tablet_service.cc:1514] Processing DeleteTablet for tablet acbf7b6c0b95402980172048cb101654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260925 04:33:34.276413 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet acbf7b6c0b95402980172048cb101654 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 04:33:34.276502 30250 tablet_replica.cc:330] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: stopping tablet replica
I20260925 04:33:34.276641 30250 raft_consensus.cc:2229] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:34.276844 30252 tablet_replica.cc:330] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: stopping tablet replica
I20260925 04:33:34.276919 30250 raft_consensus.cc:2258] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.277060 30252 raft_consensus.cc:2229] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:34.277227 30252 raft_consensus.cc:2258] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.278007 30251 tablet_replica.cc:330] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: stopping tablet replica
I20260925 04:33:34.278046 30252 ts_tablet_manager.cc:1905] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:34.278229 30251 raft_consensus.cc:2229] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:34.278386 30251 raft_consensus.cc:2258] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.279814 30250 ts_tablet_manager.cc:1905] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:34.280440 30252 ts_tablet_manager.cc:1918] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:33:34.280531 30252 log.cc:1196] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/acbf7b6c0b95402980172048cb101654
I20260925 04:33:34.280773 30252 ts_tablet_manager.cc:1939] T acbf7b6c0b95402980172048cb101654 P 09212cf3637c4ce987b9c55ef6255616: Deleting consensus metadata
I20260925 04:33:34.281322 30251 ts_tablet_manager.cc:1905] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:34.281553 29606 catalog_manager.cc:4841] TS 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383): tablet acbf7b6c0b95402980172048cb101654 (table test-table [id=17bd84d9f33244a9b142003eebbd8e2e]) successfully deleted
I20260925 04:33:34.282183 30250 ts_tablet_manager.cc:1918] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:33:34.282276 30250 log.cc:1196] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/acbf7b6c0b95402980172048cb101654
I20260925 04:33:34.282609 30250 ts_tablet_manager.cc:1939] T acbf7b6c0b95402980172048cb101654 P 47a8e56f451847618fe5a21fa16872fd: Deleting consensus metadata
I20260925 04:33:34.283233 29608 catalog_manager.cc:4841] TS 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299): tablet acbf7b6c0b95402980172048cb101654 (table test-table [id=17bd84d9f33244a9b142003eebbd8e2e]) successfully deleted
I20260925 04:33:34.283627 30251 ts_tablet_manager.cc:1918] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:33:34.283689 30251 log.cc:1196] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/acbf7b6c0b95402980172048cb101654
I20260925 04:33:34.283875 30251 ts_tablet_manager.cc:1939] T acbf7b6c0b95402980172048cb101654 P e6c2299f45d846f6b6e27c39b89defc7: Deleting consensus metadata
I20260925 04:33:34.284543 29608 catalog_manager.cc:4841] TS e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503): tablet acbf7b6c0b95402980172048cb101654 (table test-table [id=17bd84d9f33244a9b142003eebbd8e2e]) successfully deleted
Sep 25 04:33:34 dist-test-slave-r8r3 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:33:34 dist-test-slave-r8r3 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:34.313509 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:34.298678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.27.46.254:34257)
0925 04:33:34.298993 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:34.299041 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:34.299062 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:33:34.299107 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:34.299112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:34.299157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:34.299230 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:34.300010 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:34.300610 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:34.301367 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:34.301671 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:34.301765 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:34.310728 (+ 8963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:34.310730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:34.310732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:34.310755 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:34.312046 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:34.312408 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:34.312411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:34.312412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:34.312484 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:34.312726 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:34.312740 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:34.312741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:34.312894 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:34.312983 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:34.313183 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 04:33:34.313278 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20260925 04:33:34.316112 29619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:34.316583 29619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:34.321346 29763 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50a65c520e48569a9a093e663a9350 (DEFAULT_TABLE table=test-table [id=6eca13f013494b87a9f4929a6dbf5f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.321475 29912 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50a65c520e48569a9a093e663a9350 (DEFAULT_TABLE table=test-table [id=6eca13f013494b87a9f4929a6dbf5f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.321611 30056 tablet_service.cc:1467] Processing CreateTablet for tablet 0a50a65c520e48569a9a093e663a9350 (DEFAULT_TABLE table=test-table [id=6eca13f013494b87a9f4929a6dbf5f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.321702 29763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50a65c520e48569a9a093e663a9350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.321785 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50a65c520e48569a9a093e663a9350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.321875 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a50a65c520e48569a9a093e663a9350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.324376 30262 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Bootstrap starting.
I20260925 04:33:34.324568 30261 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Bootstrap starting.
I20260925 04:33:34.324590 30264 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Bootstrap starting.
I20260925 04:33:34.325301 30264 tablet_bootstrap.cc:654] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.325309 30262 tablet_bootstrap.cc:654] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.325417 30261 tablet_bootstrap.cc:654] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.326606 30262 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: No bootstrap required, opened a new log
I20260925 04:33:34.326663 30264 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: No bootstrap required, opened a new log
I20260925 04:33:34.326723 30262 ts_tablet_manager.cc:1397] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:34.326797 30264 ts_tablet_manager.cc:1397] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:34.326812 30261 tablet_bootstrap.cc:492] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: No bootstrap required, opened a new log
I20260925 04:33:34.326884 30261 ts_tablet_manager.cc:1397] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:34.327133 30262 raft_consensus.cc:348] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327250 30262 raft_consensus.cc:374] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.327219 30264 raft_consensus.cc:348] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327293 30262 raft_consensus.cc:729] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.327252 30261 raft_consensus.cc:348] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327330 30264 raft_consensus.cc:374] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.327343 30261 raft_consensus.cc:374] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.327369 30264 raft_consensus.cc:729] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.327378 30261 raft_consensus.cc:729] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.327442 30262 consensus_queue.cc:260] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327502 30261 consensus_queue.cc:260] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327502 30264 consensus_queue.cc:260] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.327737 30262 ts_tablet_manager.cc:1428] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:34.327858 30261 ts_tablet_manager.cc:1428] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:34.328179 30264 ts_tablet_manager.cc:1428] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:34.401793 30244 raft_consensus.cc:482] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:34.401918 30244 raft_consensus.cc:504] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.402406 30244 leader_election.cc:288] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:34.402902 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50a65c520e48569a9a093e663a9350" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616" is_pre_election: true
I20260925 04:33:34.402908 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50a65c520e48569a9a093e663a9350" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7" is_pre_election: true
I20260925 04:33:34.403082 29934 raft_consensus.cc:2454] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:34.403082 30078 raft_consensus.cc:2454] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:34.403443 29714 leader_election.cc:302] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09212cf3637c4ce987b9c55ef6255616, 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:34.403656 30244 raft_consensus.cc:2781] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:34.403726 30244 raft_consensus.cc:482] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:34.403754 30244 raft_consensus.cc:3037] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.404841 30244 raft_consensus.cc:504] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.405190 30244 leader_election.cc:288] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Requested vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:34.405645 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50a65c520e48569a9a093e663a9350" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616"
I20260925 04:33:34.405645 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a50a65c520e48569a9a093e663a9350" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7"
I20260925 04:33:34.405782 29934 raft_consensus.cc:3037] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.405782 30078 raft_consensus.cc:3037] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.407032 29934 raft_consensus.cc:2454] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:34.407032 30078 raft_consensus.cc:2454] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:34.407341 29716 leader_election.cc:302] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:34.407527 30244 raft_consensus.cc:2781] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:34.407639 30244 raft_consensus.cc:686] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 1 LEADER]: Becoming Leader. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Running, Role: LEADER
I20260925 04:33:34.407794 30244 consensus_queue.cc:237] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.409600 29621 catalog_manager.cc:5495] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 47a8e56f451847618fe5a21fa16872fd (127.27.46.193). New cstate: current_term: 1 leader_uuid: "47a8e56f451847618fe5a21fa16872fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:34.433609 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:34.430599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.27.46.193:37299)
0925 04:33:34.430865 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:34.430872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:34.430884 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:33:34.430921 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:34.430923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:34.430961 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:34.431043 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:34.431267 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:34.431775 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:34.432650 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:34.432867 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:34.432941 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:34.433184 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:34.433247 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:34.433297 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:34.433382 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:33:34.433422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":125,"threads_started":1}
I20260925 04:33:34.439541 29934 raft_consensus.cc:1261] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:34.439652 30078 raft_consensus.cc:1261] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:34.439983 30240 consensus_queue.cc:1035] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:34.440151 30244 consensus_queue.cc:1035] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:34.469404 29622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:34.469626 29622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:34.471638 29622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Sending DeleteTablet for 3 replicas of tablet 0a50a65c520e48569a9a093e663a9350
I20260925 04:33:34.472204 29763 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50a65c520e48569a9a093e663a9350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:58942
I20260925 04:33:34.472282 29912 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50a65c520e48569a9a093e663a9350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260925 04:33:34.472383 30250 tablet_replica.cc:330] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: stopping tablet replica
I20260925 04:33:34.472394 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a50a65c520e48569a9a093e663a9350 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:34 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 04:33:34.472447 30251 tablet_replica.cc:330] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: stopping tablet replica
I20260925 04:33:34.472498 30250 raft_consensus.cc:2229] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:34.472550 30251 raft_consensus.cc:2229] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:34.472579 30252 tablet_replica.cc:330] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: stopping tablet replica
I20260925 04:33:34.472649 30251 raft_consensus.cc:2258] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.472661 30250 raft_consensus.cc:2258] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.472695 30252 raft_consensus.cc:2229] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:34.472813 30252 raft_consensus.cc:2258] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:34.473065 29622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:33:34.473528 30251 ts_tablet_manager.cc:1905] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:34.473543 30252 ts_tablet_manager.cc:1905] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:33:34.473621 29622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:34.476140 30252 ts_tablet_manager.cc:1918] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:34.476140 30251 ts_tablet_manager.cc:1918] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:34.476212 30251 log.cc:1196] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/0a50a65c520e48569a9a093e663a9350
I20260925 04:33:34.476212 30252 log.cc:1196] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/0a50a65c520e48569a9a093e663a9350
I20260925 04:33:34.476483 30252 ts_tablet_manager.cc:1939] T 0a50a65c520e48569a9a093e663a9350 P 09212cf3637c4ce987b9c55ef6255616: Deleting consensus metadata
I20260925 04:33:34.476490 30251 ts_tablet_manager.cc:1939] T 0a50a65c520e48569a9a093e663a9350 P e6c2299f45d846f6b6e27c39b89defc7: Deleting consensus metadata
I20260925 04:33:34.477200 29608 catalog_manager.cc:4841] TS e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503): tablet 0a50a65c520e48569a9a093e663a9350 (table test-table [id=6eca13f013494b87a9f4929a6dbf5f96]) successfully deleted
I20260925 04:33:34.477197 29606 catalog_manager.cc:4841] TS 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383): tablet 0a50a65c520e48569a9a093e663a9350 (table test-table [id=6eca13f013494b87a9f4929a6dbf5f96]) successfully deleted
I20260925 04:33:34.477682 30250 ts_tablet_manager.cc:1905] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:34.478250 29763 tablet_service.cc:1467] Processing CreateTablet for tablet b98e7212b20b4bb1888008784f979080 (DEFAULT_TABLE table=test-table [id=5f02ced5915c4159b221fbff02ef8f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.478276 30056 tablet_service.cc:1467] Processing CreateTablet for tablet b98e7212b20b4bb1888008784f979080 (DEFAULT_TABLE table=test-table [id=5f02ced5915c4159b221fbff02ef8f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.478483 29763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98e7212b20b4bb1888008784f979080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.478497 30056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98e7212b20b4bb1888008784f979080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.480213 30250 ts_tablet_manager.cc:1918] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:34.480285 30250 log.cc:1196] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/0a50a65c520e48569a9a093e663a9350
I20260925 04:33:34.480301 30264 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Bootstrap starting.
I20260925 04:33:34.480554 30250 ts_tablet_manager.cc:1939] T 0a50a65c520e48569a9a093e663a9350 P 47a8e56f451847618fe5a21fa16872fd: Deleting consensus metadata
I20260925 04:33:34.480756 30262 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd: Bootstrap starting.
I20260925 04:33:34.481212 29608 catalog_manager.cc:4841] TS 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299): tablet 0a50a65c520e48569a9a093e663a9350 (table test-table [id=6eca13f013494b87a9f4929a6dbf5f96]) successfully deleted
I20260925 04:33:34.481287 30264 tablet_bootstrap.cc:654] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.481277 29912 tablet_service.cc:1467] Processing CreateTablet for tablet b98e7212b20b4bb1888008784f979080 (DEFAULT_TABLE table=test-table [id=5f02ced5915c4159b221fbff02ef8f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:34.481581 29912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98e7212b20b4bb1888008784f979080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:34.481621 30262 tablet_bootstrap.cc:654] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.482663 30264 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: No bootstrap required, opened a new log
I20260925 04:33:34.482690 30262 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd: No bootstrap required, opened a new log
I20260925 04:33:34.482723 30264 ts_tablet_manager.cc:1397] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:34.482771 30262 ts_tablet_manager.cc:1397] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 04:33:34.483013 30264 raft_consensus.cc:348] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.483115 30264 raft_consensus.cc:374] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.483136 30264 raft_consensus.cc:729] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09212cf3637c4ce987b9c55ef6255616, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.483192 30262 raft_consensus.cc:348] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.483290 30262 raft_consensus.cc:374] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.483271 30264 consensus_queue.cc:260] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.483319 30262 raft_consensus.cc:729] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.483439 30262 consensus_queue.cc:260] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.483556 30264 ts_tablet_manager.cc:1428] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:34.483743 30262 ts_tablet_manager.cc:1428] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:33:34.483981 30261 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Bootstrap starting.
I20260925 04:33:34.484792 30261 tablet_bootstrap.cc:654] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:34.486011 30261 tablet_bootstrap.cc:492] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: No bootstrap required, opened a new log
I20260925 04:33:34.486110 30261 ts_tablet_manager.cc:1397] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:34.486497 30261 raft_consensus.cc:348] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.486625 30261 raft_consensus.cc:374] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:34.486662 30261 raft_consensus.cc:729] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c2299f45d846f6b6e27c39b89defc7, State: Initialized, Role: FOLLOWER
I20260925 04:33:34.486784 30261 consensus_queue.cc:260] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.487087 30261 ts_tablet_manager.cc:1428] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:33:34.516913 30226 raft_consensus.cc:482] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:34.517028 30226 raft_consensus.cc:504] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.517519 30226 leader_election.cc:288] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:34.517897 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98e7212b20b4bb1888008784f979080" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7" is_pre_election: true
I20260925 04:33:34.518057 29934 raft_consensus.cc:2454] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:34.517999 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98e7212b20b4bb1888008784f979080" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616" is_pre_election: true
I20260925 04:33:34.518184 30078 raft_consensus.cc:2454] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 0.
I20260925 04:33:34.518401 29716 leader_election.cc:302] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8e56f451847618fe5a21fa16872fd, e6c2299f45d846f6b6e27c39b89defc7; no voters:
I20260925 04:33:34.518579 30226 raft_consensus.cc:2781] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:34.518638 30226 raft_consensus.cc:482] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:34.518664 30226 raft_consensus.cc:3037] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.519658 30226 raft_consensus.cc:504] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.520001 30226 leader_election.cc:288] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Requested vote from peers e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503), 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383)
I20260925 04:33:34.520334 29934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98e7212b20b4bb1888008784f979080" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c2299f45d846f6b6e27c39b89defc7"
I20260925 04:33:34.520342 30078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98e7212b20b4bb1888008784f979080" candidate_uuid: "47a8e56f451847618fe5a21fa16872fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09212cf3637c4ce987b9c55ef6255616"
I20260925 04:33:34.520462 29934 raft_consensus.cc:3037] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.520486 30078 raft_consensus.cc:3037] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:34.521610 29934 raft_consensus.cc:2454] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:34.521682 30078 raft_consensus.cc:2454] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8e56f451847618fe5a21fa16872fd in term 1.
I20260925 04:33:34.521996 29714 leader_election.cc:302] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09212cf3637c4ce987b9c55ef6255616, 47a8e56f451847618fe5a21fa16872fd; no voters:
I20260925 04:33:34.522171 30226 raft_consensus.cc:2781] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:34.522259 30226 raft_consensus.cc:686] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [term 1 LEADER]: Becoming Leader. State: Replica: 47a8e56f451847618fe5a21fa16872fd, State: Running, Role: LEADER
I20260925 04:33:34.522397 30226 consensus_queue.cc:237] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } }
I20260925 04:33:34.524011 29623 catalog_manager.cc:5495] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 47a8e56f451847618fe5a21fa16872fd (127.27.46.193). New cstate: current_term: 1 leader_uuid: "47a8e56f451847618fe5a21fa16872fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8e56f451847618fe5a21fa16872fd" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:34.544366 30078 raft_consensus.cc:1261] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:34.544461 29933 raft_consensus.cc:1261] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Refusing update from remote peer 47a8e56f451847618fe5a21fa16872fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:34.544807 30226 consensus_queue.cc:1035] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "09212cf3637c4ce987b9c55ef6255616" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:34.544958 30244 consensus_queue.cc:1035] T b98e7212b20b4bb1888008784f979080 P 47a8e56f451847618fe5a21fa16872fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c2299f45d846f6b6e27c39b89defc7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:33:34.558323 30231 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:33:36.107715 29621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:36.107860 29621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:36.110172 29621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944613e8e32b4775bcb0582e223a98e6: Sending DeleteTablet for 3 replicas of tablet b98e7212b20b4bb1888008784f979080
I20260925 04:33:36.110734 30056 tablet_service.cc:1514] Processing DeleteTablet for tablet b98e7212b20b4bb1888008784f979080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:36 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40372
I20260925 04:33:36.110737 29912 tablet_service.cc:1514] Processing DeleteTablet for tablet b98e7212b20b4bb1888008784f979080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:36 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260925 04:33:36.111065 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29697
I20260925 04:33:36.111058 29763 tablet_service.cc:1514] Processing DeleteTablet for tablet b98e7212b20b4bb1888008784f979080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:36 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:58942
I20260925 04:33:36.111101 30309 tablet_replica.cc:330] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: stopping tablet replica
I20260925 04:33:36.112082 30309 raft_consensus.cc:2229] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:36.112241 30309 raft_consensus.cc:2258] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:36.112486 30308 tablet_replica.cc:330] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: stopping tablet replica
I20260925 04:33:36.112660 30308 raft_consensus.cc:2229] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:36.112788 30308 raft_consensus.cc:2258] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:36.112927 30309 ts_tablet_manager.cc:1905] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:36.113322 30308 ts_tablet_manager.cc:1905] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:36.115718 30309 ts_tablet_manager.cc:1918] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:33:36.115811 30309 log.cc:1196] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/b98e7212b20b4bb1888008784f979080
I20260925 04:33:36.116106 30309 ts_tablet_manager.cc:1939] T b98e7212b20b4bb1888008784f979080 P e6c2299f45d846f6b6e27c39b89defc7: Deleting consensus metadata
I20260925 04:33:36.116643 30308 ts_tablet_manager.cc:1918] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:33:36.116760 30308 log.cc:1196] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/b98e7212b20b4bb1888008784f979080
I20260925 04:33:36.116743 29608 catalog_manager.cc:4841] TS e6c2299f45d846f6b6e27c39b89defc7 (127.27.46.194:42503): tablet b98e7212b20b4bb1888008784f979080 (table test-table [id=5f02ced5915c4159b221fbff02ef8f55]) successfully deleted
I20260925 04:33:36.117048 30308 ts_tablet_manager.cc:1939] T b98e7212b20b4bb1888008784f979080 P 09212cf3637c4ce987b9c55ef6255616: Deleting consensus metadata
I20260925 04:33:36.117728 29606 catalog_manager.cc:4841] TS 09212cf3637c4ce987b9c55ef6255616 (127.27.46.195:36383): tablet b98e7212b20b4bb1888008784f979080 (table test-table [id=5f02ced5915c4159b221fbff02ef8f55]) successfully deleted
W20260925 04:33:36.120961 29608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:37299 (error 108)
I20260925 04:33:36.121165 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29844
I20260925 04:33:36.122970 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:36.122454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:37299 (local address 127.0.0.1:46022)
0925 04:33:36.122732 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:36.122847 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:37299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
W20260925 04:33:36.123087 29608 catalog_manager.cc:4584] TS 47a8e56f451847618fe5a21fa16872fd (127.27.46.193:37299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b98e7212b20b4bb1888008784f979080: Network error: Client connection negotiation failed: client connection to 127.27.46.193:37299: connect: Connection refused (error 111)
I20260925 04:33:36.128849 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29990
I20260925 04:33:36.135766 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 29589
2026-09-25T04:33:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6212 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:36 dist-test-slave-r8r3 krb5kdc[30316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:36 dist-test-slave-r8r3 krb5kdc[30316](info): set up 2 sockets
Sep 25 04:33:36 dist-test-slave-r8r3 krb5kdc[30316](info): commencing operation
krb5kdc: starting...
W20260925 04:33:37.165181 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:37.284066 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44917
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:37.391232 30330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:37.391499 30330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:37.391567 30330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:37.395188 30330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:37.395274 30330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:37.395309 30330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:37.395364 30330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:37.395394 30330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:37.400216 30330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:37.401260 30330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:37.402251 30330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:37.410712 30339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:37.410722 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:37.410774 30330 server_base.cc:1034] running on GCE node
W20260925 04:33:37.410729 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:37.411446 30330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:37.412514 30330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:37.413666 30330 hybrid_clock.cc:648] HybridClock initialized: now 1790310817413645 us; error 30 us; skew 500 ppm
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:37.416565 30330 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:37.417579 30330 webserver.cc:460] Webserver started at http://127.27.46.254:46575/ using document root <none> and password file <none>
I20260925 04:33:37.418141 30330 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:37.418205 30330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:37.418439 30330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:37.420466 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "7dd458778b50412ca934c3a22609838f"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "803c2ad6656049afffccc064ce4e4c2c"
server_key_iv: "600a9cae3c9f3752d47a9c1fb35178f1"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.421043 30330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "7dd458778b50412ca934c3a22609838f"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "803c2ad6656049afffccc064ce4e4c2c"
server_key_iv: "600a9cae3c9f3752d47a9c1fb35178f1"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.424646 30330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:37.427040 30346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.428254 30330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:37.428406 30330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "7dd458778b50412ca934c3a22609838f"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "803c2ad6656049afffccc064ce4e4c2c"
server_key_iv: "600a9cae3c9f3752d47a9c1fb35178f1"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.428517 30330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:37.449131 30330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:37.452380 30330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:37.452595 30330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:37.460261 30330 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:44917
I20260925 04:33:37.460258 30398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:44917 every 8 connection(s)
I20260925 04:33:37.461222 30330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:37.463953 30399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:37.469215 30399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f: Bootstrap starting.
I20260925 04:33:37.470523 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30330
I20260925 04:33:37.470685 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:37.470970 27835 external_mini_cluster.cc:1428] Setting key aa1600fc4f4a6385d5e6ea4ee4646606
I20260925 04:33:37.471853 30399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:37.472599 30399 log.cc:824] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:37.474856 30399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f: No bootstrap required, opened a new log
I20260925 04:33:37.479175 30399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } }
I20260925 04:33:37.479490 30399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:37.479562 30399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dd458778b50412ca934c3a22609838f, State: Initialized, Role: FOLLOWER
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:37.480036 30399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } }
I20260925 04:33:37.480186 30399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:37.480258 30399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:37.480371 30399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:37.481280 30399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } }
I20260925 04:33:37.481657 30399 leader_election.cc:302] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dd458778b50412ca934c3a22609838f; no voters:
I20260925 04:33:37.482003 30399 leader_election.cc:288] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:37.482091 30404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:37.482302 30404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [term 1 LEADER]: Becoming Leader. State: Replica: 7dd458778b50412ca934c3a22609838f, State: Running, Role: LEADER
I20260925 04:33:37.482638 30404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } }
I20260925 04:33:37.483253 30399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:37.484400 30405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dd458778b50412ca934c3a22609838f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } } }
I20260925 04:33:37.484515 30405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:37.484915 30406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dd458778b50412ca934c3a22609838f. Latest consensus state: current_term: 1 leader_uuid: "7dd458778b50412ca934c3a22609838f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dd458778b50412ca934c3a22609838f" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44917 } } }
I20260925 04:33:37.485077 30406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:37.485860 30413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:37.486721 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:37.472088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.27.46.254:44917)
0925 04:33:37.472550 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:37.472567 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:37.472604 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:33:37.473542 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:37.473568 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:37.473911 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:37.474208 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:37.475149 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.476114 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.477322 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.477843 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.477961 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:37.480714 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:37.480718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:37.480730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:37.480757 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:37.482771 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.483257 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.483265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.483274 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.483411 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.483816 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.483819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.483821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.484373 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:37.485350 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:37.485529 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 04:33:37.485854 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260925 04:33:37.489215 30413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:37.494042 30413 catalog_manager.cc:1347] Generated new cluster ID: 6adfda781be54605bc42fd4e67784305
I20260925 04:33:37.494102 30413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:37.503664 30413 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:37.504640 30413 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:37.512437 30413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dd458778b50412ca934c3a22609838f: Generated new TSK 0
I20260925 04:33:37.513033 30413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:37.554697 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:37.665058 30425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:37.665311 30425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:37.665351 30425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:37.669359 30425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:37.669418 30425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:37.669474 30425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:37.674430 30425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:37.675540 30425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:37.676620 30425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:37.686514 30434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:37.686514 30432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:37.686514 30430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:37.686926 30425 server_base.cc:1034] running on GCE node
I20260925 04:33:37.687392 30425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:37.688079 30425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:37.689244 30425 hybrid_clock.cc:648] HybridClock initialized: now 1790310817689227 us; error 30 us; skew 500 ppm
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:37.692081 30425 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:37.693013 30425 webserver.cc:460] Webserver started at http://127.27.46.193:39581/ using document root <none> and password file <none>
I20260925 04:33:37.693660 30425 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:37.693768 30425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:37.694005 30425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:37.696091 30425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "333fe3b57f8344fcbab3cb7fb7727ac5"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "ab698d7828f5a1566538dc7120e44531"
server_key_iv: "79119881aaff35f621d613e32ecf7480"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.696729 30425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "333fe3b57f8344fcbab3cb7fb7727ac5"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "ab698d7828f5a1566538dc7120e44531"
server_key_iv: "79119881aaff35f621d613e32ecf7480"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.700587 30425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:33:37.703130 30441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.704578 30425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:33:37.704711 30425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "333fe3b57f8344fcbab3cb7fb7727ac5"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "ab698d7828f5a1566538dc7120e44531"
server_key_iv: "79119881aaff35f621d613e32ecf7480"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.704825 30425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:37.718288 30425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:37.721086 30425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:37.721285 30425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:37.721971 30425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:37.722811 30425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:37.722882 30425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.722954 30425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:37.722996 30425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.731916 30425 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:32997
I20260925 04:33:37.731940 30560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:32997 every 8 connection(s)
I20260925 04:33:37.732834 30425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:37.741120 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30425
I20260925 04:33:37.741255 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:37.741628 27835 external_mini_cluster.cc:1428] Setting key 8143a75202df8b7c4f12f65b0ace6f1b
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:37.745934 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:37.734635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57073 (local address 127.27.46.254:44917)
0925 04:33:37.734789 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:37.734792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:37.735751 (+ 959us) server_negotiation.cc:407] Connection header received
0925 04:33:37.736617 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:37.736620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:37.736671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:37.736748 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:37.737939 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.738472 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.739531 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.739758 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.739834 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:37.742397 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:37.742401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:37.742403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:37.742430 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:37.743860 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.744364 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.744368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.744369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.744456 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.744795 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.744799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.744801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.744989 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:37.745104 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:37.745638 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 04:33:37.745764 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:33:37.746740 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:37.734972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44917 (local address 127.27.46.193:57073)
0925 04:33:37.735607 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:37.735647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:37.736396 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:37.736912 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:37.736919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:37.737261 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:37.737803 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:37.737817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.738577 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.738579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:37.739426 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:37.739435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.739886 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.739889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:37.740020 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:37.740610 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:37.740627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:37.742222 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:37.743979 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:37.743985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:37.744001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:37.744253 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:37.744550 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:37.744554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:37.744557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:37.744696 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:37.745100 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:37.745106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:37.745353 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 04:33:37.745596 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 04:33:37.745912 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":111,"threads_started":1}
I20260925 04:33:37.747879 30561 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44917
I20260925 04:33:37.748152 30561 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:37.748623 30561 heartbeater.cc:511] Master 127.27.46.254:44917 requested a full tablet report, sending...
I20260925 04:33:37.750145 30363 ts_manager.cc:194] Registered new tserver with Master: 333fe3b57f8344fcbab3cb7fb7727ac5 (127.27.46.193:32997)
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:33:37.751325 30363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:57073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:37.767884 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:37.875919 30566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:37.876235 30566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:37.876330 30566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:37.880270 30566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:37.880362 30566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:37.880450 30566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:37.885756 30566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:37.886781 30566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:37.887821 30566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:37.897347 30571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:37.897548 30566 server_base.cc:1034] running on GCE node
W20260925 04:33:37.897572 30574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:37.897369 30572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:37.898123 30566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:37.898690 30566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:37.899845 30566 hybrid_clock.cc:648] HybridClock initialized: now 1790310817899833 us; error 24 us; skew 500 ppm
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:37.902554 30566 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:37.903414 30566 webserver.cc:460] Webserver started at http://127.27.46.194:35483/ using document root <none> and password file <none>
I20260925 04:33:37.903976 30566 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:37.904074 30566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:37.904299 30566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:37.906319 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "75d82542be8a418b8bc7a176bdf1ff0b"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "890175bbdbe1cabe993fc74a6551ae4b"
server_key_iv: "b198526295e40660fc37d0492683ec9b"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.906936 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "75d82542be8a418b8bc7a176bdf1ff0b"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "890175bbdbe1cabe993fc74a6551ae4b"
server_key_iv: "b198526295e40660fc37d0492683ec9b"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.910521 30566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 04:33:37.912756 30581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.914033 30566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:37.914160 30566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "75d82542be8a418b8bc7a176bdf1ff0b"
format_stamp: "Formatted at 2026-09-25 04:33:37 on dist-test-slave-r8r3"
server_key: "890175bbdbe1cabe993fc74a6551ae4b"
server_key_iv: "b198526295e40660fc37d0492683ec9b"
server_key_version: "encryptionkey@0"
I20260925 04:33:37.914263 30566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:37.951036 30566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:37.954010 30566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:37.954188 30566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:37.954766 30566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:37.955582 30566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:37.955650 30566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.955722 30566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:37.955765 30566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:37.964962 30566 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:39587
I20260925 04:33:37.964982 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:39587 every 8 connection(s)
I20260925 04:33:37.965937 30566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:37.974408 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30566
I20260925 04:33:37.974511 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:37.974784 27835 external_mini_cluster.cc:1428] Setting key a32b5f91f1cbe094b315ed604f7b8461
Sep 25 04:33:37 dist-test-slave-r8r3 krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:37.979442 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:37.967758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:57661 (local address 127.27.46.254:44917)
0925 04:33:37.967922 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:37.967927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:37.969030 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 04:33:37.969919 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:37.969922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:37.969969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:37.970049 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:37.971157 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.971683 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.972811 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.973029 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.973119 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:37.975665 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:37.975668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:37.975670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:37.975693 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:37.977155 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.977774 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.977778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.977779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.977855 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:37.978216 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:37.978219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:37.978221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:37.978487 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:37.978602 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:37.979140 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 04:33:37.979271 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:33:37.980865 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:37.968209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44917 (local address 127.27.46.194:57661)
0925 04:33:37.968870 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:37.968911 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:37.969729 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:37.970217 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:37.970223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:37.970560 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:37.971025 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:37.971037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.971799 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.971802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:37.972681 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:37.972690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:37.973170 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:37.973173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:37.973308 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:37.973982 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:37.974000 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:37.975504 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:37.977282 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:37.977288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:37.977303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:37.977658 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:37.977953 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:37.977957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:37.977959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:37.978112 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:37.978625 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:37.978632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:37.978884 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 04:33:37.979073 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 04:33:37.979414 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":121,"threads_started":1}
I20260925 04:33:37.982581 30701 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44917
I20260925 04:33:37.982883 30701 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:37.983417 30701 heartbeater.cc:511] Master 127.27.46.254:44917 requested a full tablet report, sending...
I20260925 04:33:37.984560 30363 ts_manager.cc:194] Registered new tserver with Master: 75d82542be8a418b8bc7a176bdf1ff0b (127.27.46.194:39587)
I20260925 04:33:37.985255 30363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:57661
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:38.004284 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:38.114295 30706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:38.114552 30706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:38.114593 30706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:38.118605 30706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:38.118657 30706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:38.118713 30706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:38.123625 30706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:38.124579 30706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:38.125514 30706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:38.135013 30712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:38.135102 30711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:38.135036 30715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:38.135298 30706 server_base.cc:1034] running on GCE node
I20260925 04:33:38.135689 30706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:38.136273 30706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:38.137437 30706 hybrid_clock.cc:648] HybridClock initialized: now 1790310818137423 us; error 24 us; skew 500 ppm
Sep 25 04:33:38 dist-test-slave-r8r3 krb5kdc[30316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310818, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:38.140079 30706 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:38.140919 30706 webserver.cc:460] Webserver started at http://127.27.46.195:35373/ using document root <none> and password file <none>
I20260925 04:33:38.141566 30706 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:38.141675 30706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:38.141906 30706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:38.143944 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "89624c8a272d43d1aeffff4a5c6ae9b9"
format_stamp: "Formatted at 2026-09-25 04:33:38 on dist-test-slave-r8r3"
server_key: "62e368184d3f1b49271b84f9c12c9499"
server_key_iv: "7e5a20d8b642e41bdc163bf0b51503d5"
server_key_version: "encryptionkey@0"
I20260925 04:33:38.144511 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "89624c8a272d43d1aeffff4a5c6ae9b9"
format_stamp: "Formatted at 2026-09-25 04:33:38 on dist-test-slave-r8r3"
server_key: "62e368184d3f1b49271b84f9c12c9499"
server_key_iv: "7e5a20d8b642e41bdc163bf0b51503d5"
server_key_version: "encryptionkey@0"
I20260925 04:33:38.149047 30706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 04:33:38.151470 30723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:38.152649 30706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:38.152770 30706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "89624c8a272d43d1aeffff4a5c6ae9b9"
format_stamp: "Formatted at 2026-09-25 04:33:38 on dist-test-slave-r8r3"
server_key: "62e368184d3f1b49271b84f9c12c9499"
server_key_iv: "7e5a20d8b642e41bdc163bf0b51503d5"
server_key_version: "encryptionkey@0"
I20260925 04:33:38.152863 30706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:38.172650 30706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:38.175513 30706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:38.175719 30706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:38.176331 30706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:38.177152 30706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:38.177196 30706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:38.177263 30706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:38.177304 30706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:38.186424 30706 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:46037
I20260925 04:33:38.186450 30838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:46037 every 8 connection(s)
I20260925 04:33:38.187304 30706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:38.190786 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30706
I20260925 04:33:38.190943 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:38.191282 27835 external_mini_cluster.cc:1428] Setting key 48c94232671531630d31aed3eb06beb3
Sep 25 04:33:38 dist-test-slave-r8r3 krb5kdc[30316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310818, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:38.200172 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:38.189174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:40231 (local address 127.27.46.254:44917)
0925 04:33:38.189325 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:38.189329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:38.190269 (+ 940us) server_negotiation.cc:407] Connection header received
0925 04:33:38.191206 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:38.191210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:38.191270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:38.191364 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:38.192727 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:38.193229 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:38.194303 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:38.194518 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:38.194617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:38.196893 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:38.196896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:38.196899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:38.196925 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:38.198242 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:38.198737 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:38.198741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:38.198743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:38.198822 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:38.199161 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:38.199164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:38.199166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:38.199352 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:38.199469 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:38.199910 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 04:33:38.200017 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:33:38.201000 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:38.189465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44917 (local address 127.27.46.195:40231)
0925 04:33:38.190107 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:38.190147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:38.190949 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:38.191538 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:38.191545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:38.191998 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:38.192582 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:38.192601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:38.193349 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:38.193352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:38.194180 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:38.194189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:38.194666 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:38.194669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:38.194819 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:38.195402 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:38.195418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:38.196737 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:38.198359 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:38.198365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:38.198379 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:38.198630 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:38.198923 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:38.198926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:38.198928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:38.199060 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:38.199469 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:38.199475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:38.199695 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 04:33:38.199870 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 04:33:38.200170 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":98,"threads_started":1}
I20260925 04:33:38.202117 30839 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44917
I20260925 04:33:38.202340 30839 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:38.202780 30839 heartbeater.cc:511] Master 127.27.46.254:44917 requested a full tablet report, sending...
I20260925 04:33:38.203785 30363 ts_manager.cc:194] Registered new tserver with Master: 89624c8a272d43d1aeffff4a5c6ae9b9 (127.27.46.195:46037)
I20260925 04:33:38.204470 30363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:40231
I20260925 04:33:38.204779 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:33:38.217342 30402 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:38.216074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.27.46.254:44917)
0925 04:33:38.216204 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:38.216208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:38.216375 (+ 167us) server_negotiation.cc:407] Connection header received
0925 04:33:38.216474 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:38.216477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:38.216513 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:38.216580 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:38.217250 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46378: BlockingRecv error: recv got EOF from 127.0.0.1:46378 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:38.218667 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30425
I20260925 04:33:38.224727 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30566
I20260925 04:33:38.230202 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30706
I20260925 04:33:38.235468 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30330
2026-09-25T04:33:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2099 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:38 dist-test-slave-r8r3 krb5kdc[30852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:38 dist-test-slave-r8r3 krb5kdc[30852](info): set up 2 sockets
Sep 25 04:33:38 dist-test-slave-r8r3 krb5kdc[30852](info): commencing operation
krb5kdc: starting...
W20260925 04:33:39.245025 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:39.360486 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44497
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:39.465811 30865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:39.466094 30865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:39.466163 30865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:39.470048 30865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:39.470144 30865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:39.470176 30865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:39.470224 30865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:39.470259 30865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:39.475152 30865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:39.476202 30865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:39.477218 30865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:39.485564 30870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:39.485567 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:39.485666 30865 server_base.cc:1034] running on GCE node
W20260925 04:33:39.485657 30871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:39.486493 30865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:39.487536 30865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:39.488723 30865 hybrid_clock.cc:648] HybridClock initialized: now 1790310819488709 us; error 35 us; skew 500 ppm
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:39.497228 30865 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:39.498204 30865 webserver.cc:460] Webserver started at http://127.27.46.254:46785/ using document root <none> and password file <none>
I20260925 04:33:39.498772 30865 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:39.498867 30865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:39.499111 30865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:39.501163 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "b451edad26df44f2bcce6fa1c67a4f8a"
format_stamp: "Formatted at 2026-09-25 04:33:39 on dist-test-slave-r8r3"
server_key: "69c95c967638d3f4ae6c2bd0aed68b99"
server_key_iv: "f4cc7edfc6175a55f19802d30590be43"
server_key_version: "encryptionkey@0"
I20260925 04:33:39.501760 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "b451edad26df44f2bcce6fa1c67a4f8a"
format_stamp: "Formatted at 2026-09-25 04:33:39 on dist-test-slave-r8r3"
server_key: "69c95c967638d3f4ae6c2bd0aed68b99"
server_key_iv: "f4cc7edfc6175a55f19802d30590be43"
server_key_version: "encryptionkey@0"
I20260925 04:33:39.505457 30865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:33:39.507885 30880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:39.509127 30865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:39.509267 30865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "b451edad26df44f2bcce6fa1c67a4f8a"
format_stamp: "Formatted at 2026-09-25 04:33:39 on dist-test-slave-r8r3"
server_key: "69c95c967638d3f4ae6c2bd0aed68b99"
server_key_iv: "f4cc7edfc6175a55f19802d30590be43"
server_key_version: "encryptionkey@0"
I20260925 04:33:39.509390 30865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:39.521782 30865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:39.524587 30865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:39.524797 30865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:39.531821 30865 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:44497
I20260925 04:33:39.532101 30932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:44497 every 8 connection(s)
I20260925 04:33:39.532766 30865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:39.535416 30933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:39.536409 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30865
I20260925 04:33:39.536525 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:39.536877 27835 external_mini_cluster.cc:1428] Setting key 43e376bc5c12f9de844601fa84fca1b3
I20260925 04:33:39.541924 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a: Bootstrap starting.
I20260925 04:33:39.544701 30933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:39.545643 30933 log.cc:824] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:39.548099 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a: No bootstrap required, opened a new log
I20260925 04:33:39.551689 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:39.538163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.27.46.254:44497)
0925 04:33:39.538568 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:39.538585 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:39.538620 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:33:39.539255 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:39.539271 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:39.539591 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:39.539843 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:39.540749 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:39.541773 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:39.542951 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:39.543426 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:39.543534 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:39.546583 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:39.546589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:39.546604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:39.546631 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:39.548626 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:39.549035 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:39.549042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:39.549047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:39.549181 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:39.549453 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:39.549456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:39.549458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:39.549976 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:39.550191 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:39.550410 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:33:39.550758 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":134,"threads_started":1}
I20260925 04:33:39.552582 30933 raft_consensus.cc:348] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } }
I20260925 04:33:39.552816 30933 raft_consensus.cc:374] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:39.552901 30933 raft_consensus.cc:729] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b451edad26df44f2bcce6fa1c67a4f8a, State: Initialized, Role: FOLLOWER
I20260925 04:33:39.553363 30933 consensus_queue.cc:260] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } }
I20260925 04:33:39.553542 30933 raft_consensus.cc:388] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:39.553639 30933 raft_consensus.cc:482] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:39.553745 30933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:39.554600 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:33:39.554682 30933 raft_consensus.cc:504] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } }
I20260925 04:33:39.555065 30933 leader_election.cc:302] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b451edad26df44f2bcce6fa1c67a4f8a; no voters:
I20260925 04:33:39.555409 30933 leader_election.cc:288] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:39.555490 30938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:39.555686 30938 raft_consensus.cc:686] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [term 1 LEADER]: Becoming Leader. State: Replica: b451edad26df44f2bcce6fa1c67a4f8a, State: Running, Role: LEADER
I20260925 04:33:39.555997 30938 consensus_queue.cc:237] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } }
I20260925 04:33:39.556649 30933 sys_catalog.cc:564] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:39.557114 30939 sys_catalog.cc:455] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } } }
I20260925 04:33:39.557225 30939 sys_catalog.cc:458] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:39.557520 30940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b451edad26df44f2bcce6fa1c67a4f8a. Latest consensus state: current_term: 1 leader_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b451edad26df44f2bcce6fa1c67a4f8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44497 } } }
I20260925 04:33:39.557624 30940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:39.558048 30943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:39.561906 30943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:39.567447 30943 catalog_manager.cc:1347] Generated new cluster ID: 6945322a459b431d99125c0712f15e5b
I20260925 04:33:39.567507 30943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:39.575104 30943 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:39.576313 30943 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:39.578513 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:39.573225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.27.46.254:44497)
0925 04:33:39.573353 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:39.573358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:39.573667 (+ 309us) server_negotiation.cc:407] Connection header received
0925 04:33:39.573838 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:39.573863 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:39.573915 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:39.573997 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:39.574654 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:39.575393 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:39.576256 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:39.576564 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:39.576677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:39.577569 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:39.577573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:39.577576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:39.577984 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:39.578012 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:39.578028 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:39.578137 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:39.578293 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 04:33:39.578371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:33:39.585328 30943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b451edad26df44f2bcce6fa1c67a4f8a: Generated new TSK 0
I20260925 04:33:39.586084 30943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:33:39.591030 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30865
2026-09-25T04:33:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1352 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30968](info): set up 2 sockets
Sep 25 04:33:39 dist-test-slave-r8r3 krb5kdc[30968](info): commencing operation
krb5kdc: starting...
W20260925 04:33:40.604746 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:40 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:40.722766 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35363
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:40.830104 30982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:40.830406 30982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:40.830494 30982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:40.834476 30982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:40.834569 30982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:40.834609 30982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:40.834668 30982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:40.834709 30982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:40.839855 30982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:40.840890 30982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:40.841905 30982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:40.850437 30989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:40.850523 30982 server_base.cc:1034] running on GCE node
W20260925 04:33:40.850450 30988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:40.850454 30991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:40.851282 30982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:40.852388 30982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:40.853585 30982 hybrid_clock.cc:648] HybridClock initialized: now 1790310820853546 us; error 46 us; skew 500 ppm
Sep 25 04:33:40 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310820, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:40.856245 30982 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:40.857093 30982 webserver.cc:460] Webserver started at http://127.27.46.254:40683/ using document root <none> and password file <none>
I20260925 04:33:40.857671 30982 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:40.857736 30982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:40.857896 30982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:40.859877 30982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "547a81438e554e83b225a84634213953"
format_stamp: "Formatted at 2026-09-25 04:33:40 on dist-test-slave-r8r3"
server_key: "99cb2875bdcef8c4963b288af266d92a"
server_key_iv: "dc89578c96f9d9899ef3a49f7e210be8"
server_key_version: "encryptionkey@0"
I20260925 04:33:40.860374 30982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "547a81438e554e83b225a84634213953"
format_stamp: "Formatted at 2026-09-25 04:33:40 on dist-test-slave-r8r3"
server_key: "99cb2875bdcef8c4963b288af266d92a"
server_key_iv: "dc89578c96f9d9899ef3a49f7e210be8"
server_key_version: "encryptionkey@0"
I20260925 04:33:40.864169 30982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:33:40.866546 30998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:40.867786 30982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:40.867933 30982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "547a81438e554e83b225a84634213953"
format_stamp: "Formatted at 2026-09-25 04:33:40 on dist-test-slave-r8r3"
server_key: "99cb2875bdcef8c4963b288af266d92a"
server_key_iv: "dc89578c96f9d9899ef3a49f7e210be8"
server_key_version: "encryptionkey@0"
I20260925 04:33:40.868032 30982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:40.880550 30982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:40.883836 30982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:40.884044 30982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:40.891115 30982 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35363
I20260925 04:33:40.891387 31052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35363 every 8 connection(s)
I20260925 04:33:40.892115 30982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:40.894856 31053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:40.898790 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 30982
I20260925 04:33:40.898912 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:40.899217 27835 external_mini_cluster.cc:1428] Setting key b3e1025f97e4d2eebc1102a0d84cf300
I20260925 04:33:40.900848 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953: Bootstrap starting.
I20260925 04:33:40.903414 31053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:40.904184 31053 log.cc:824] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:40.906410 31053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953: No bootstrap required, opened a new log
Sep 25 04:33:40 dist-test-slave-r8r3 krb5kdc[30968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:40.910724 31053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } }
I20260925 04:33:40.910980 31053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:40.911078 31053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547a81438e554e83b225a84634213953, State: Initialized, Role: FOLLOWER
I20260925 04:33:40.911571 31053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } }
I20260925 04:33:40.911727 31053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:40.911815 31053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:40.911919 31053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:40.912863 31053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } }
I20260925 04:33:40.913235 31053 leader_election.cc:302] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547a81438e554e83b225a84634213953; no voters:
I20260925 04:33:40.913719 31053 leader_election.cc:288] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:40.913859 31059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:40.914085 31059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [term 1 LEADER]: Becoming Leader. State: Replica: 547a81438e554e83b225a84634213953, State: Running, Role: LEADER
I20260925 04:33:40.914382 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:40.900612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.27.46.254:35363)
0925 04:33:40.901027 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:40.901043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:40.901086 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:33:40.901775 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:40.901792 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:40.902126 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:40.902397 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:40.903296 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:40.904402 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:40.905479 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:40.905986 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:40.906114 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:40.908888 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:40.908893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:40.908906 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:40.908933 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:40.910801 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:40.911330 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:40.911339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:40.911348 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:40.911495 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:40.911813 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:40.911819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:40.911821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:40.912324 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:40.912534 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:40.912844 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 04:33:40.913174 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260925 04:33:40.914821 31059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } }
I20260925 04:33:40.915303 31053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:40.916812 31061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547a81438e554e83b225a84634213953. Latest consensus state: current_term: 1 leader_uuid: "547a81438e554e83b225a84634213953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } } }
I20260925 04:33:40.917256 31061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:40.917130 31060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547a81438e554e83b225a84634213953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547a81438e554e83b225a84634213953" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35363 } } }
I20260925 04:33:40.917596 31060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:40.917709 31073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:40.920650 31073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:40.925920 31073 catalog_manager.cc:1347] Generated new cluster ID: e5887e9ce77240f0ae48e9a04a80b760
I20260925 04:33:40.925981 31073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:40.944281 31073 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:40.945272 31073 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:40.959219 31073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547a81438e554e83b225a84634213953: Generated new TSK 0
I20260925 04:33:40.959947 31073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:40.992558 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:41.121767 31080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:41.122063 31080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:41.122151 31080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:41.126276 31080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:41.126354 31080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:41.126443 31080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:41.131608 31080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:41.132597 31080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:41.133764 31080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:41.142949 31087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:41.143085 31090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:41.142982 31088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:41.143265 31080 server_base.cc:1034] running on GCE node
I20260925 04:33:41.143632 31080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:41.144284 31080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:41.145480 31080 hybrid_clock.cc:648] HybridClock initialized: now 1790310821145459 us; error 31 us; skew 500 ppm
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:41.148303 31080 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:41.149199 31080 webserver.cc:460] Webserver started at http://127.27.46.193:41699/ using document root <none> and password file <none>
I20260925 04:33:41.149816 31080 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:41.149881 31080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:41.150142 31080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:41.152029 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "d06f56f3e1a44de78cb061cbcba0f89a"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "659501218937380b5a94f347da2556a5"
server_key_iv: "10c40cdb318b39bf59121ae881814f09"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.152581 31080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "d06f56f3e1a44de78cb061cbcba0f89a"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "659501218937380b5a94f347da2556a5"
server_key_iv: "10c40cdb318b39bf59121ae881814f09"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.156364 31080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260925 04:33:41.159547 31097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.160872 31080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:33:41.160959 31080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "d06f56f3e1a44de78cb061cbcba0f89a"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "659501218937380b5a94f347da2556a5"
server_key_iv: "10c40cdb318b39bf59121ae881814f09"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.161041 31080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:41.184585 31080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:41.187449 31080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:41.187650 31080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:41.188234 31080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:41.189050 31080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:41.189100 31080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.189162 31080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:41.189193 31080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.197453 31080 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:43653
I20260925 04:33:41.197482 31214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:43653 every 8 connection(s)
I20260925 04:33:41.198308 31080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:41.199445 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31080
I20260925 04:33:41.199579 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:41.199887 27835 external_mini_cluster.cc:1428] Setting key 4fbf2b0ba31d122170bed96df00f7c8f
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:41.212447 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.200251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57003 (local address 127.27.46.254:35363)
0925 04:33:41.200393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:41.200396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:41.201448 (+ 1052us) server_negotiation.cc:407] Connection header received
0925 04:33:41.202769 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:41.202772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:41.202824 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:41.202905 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:41.204052 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.204596 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.205869 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.206100 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.206203 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.208555 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:41.208558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:41.208560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:41.208585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:41.210154 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.210641 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.210645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.210647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.210758 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.211075 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.211077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.211078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.211328 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:41.211435 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:41.212036 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 04:33:41.212183 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:33:41.213135 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.200732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35363 (local address 127.27.46.193:57003)
0925 04:33:41.201260 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:41.201334 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:41.202577 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:41.203091 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:41.203098 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:41.203427 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:41.203908 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.203922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.204722 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.204725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.205742 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.205752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.206251 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.206256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.206420 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.206988 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:41.207003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:41.208385 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:41.210266 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.210271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.210282 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.210527 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.210852 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.210854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.210856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.210988 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.211436 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:41.211441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:41.211762 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 04:33:41.211944 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:33:41.212224 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260925 04:33:41.214579 31215 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35363
I20260925 04:33:41.214831 31215 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:41.215415 31215 heartbeater.cc:511] Master 127.27.46.254:35363 requested a full tablet report, sending...
I20260925 04:33:41.217236 31016 ts_manager.cc:194] Registered new tserver with Master: d06f56f3e1a44de78cb061cbcba0f89a (127.27.46.193:43653)
I20260925 04:33:41.218425 31016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:57003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:41.227794 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:41.337633 31220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:41.337904 31220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:41.337975 31220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:41.342007 31220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:41.342092 31220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:41.342176 31220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:41.347424 31220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:41.348483 31220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:41.349587 31220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:41.359160 31227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:41.359186 31229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:41.359146 31226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:41.359390 31220 server_base.cc:1034] running on GCE node
I20260925 04:33:41.359867 31220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:41.360507 31220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:41.361714 31220 hybrid_clock.cc:648] HybridClock initialized: now 1790310821361666 us; error 59 us; skew 500 ppm
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:41.364509 31220 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:41.365437 31220 webserver.cc:460] Webserver started at http://127.27.46.194:37685/ using document root <none> and password file <none>
I20260925 04:33:41.366072 31220 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:41.366165 31220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:41.366402 31220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:41.368464 31220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "80174a458bbd4f0793ea6ff83dd19d0b"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "3eeaba4bac49edb4025d30b0ae3daaa6"
server_key_iv: "fb5372fb55ab512c662cfc60f1596c7c"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.369046 31220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "80174a458bbd4f0793ea6ff83dd19d0b"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "3eeaba4bac49edb4025d30b0ae3daaa6"
server_key_iv: "fb5372fb55ab512c662cfc60f1596c7c"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.372880 31220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:33:41.375386 31236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.376629 31220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:33:41.376768 31220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "80174a458bbd4f0793ea6ff83dd19d0b"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "3eeaba4bac49edb4025d30b0ae3daaa6"
server_key_iv: "fb5372fb55ab512c662cfc60f1596c7c"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.376904 31220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:41.387658 31220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:41.390400 31220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:41.390618 31220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:41.391229 31220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:41.392050 31220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:41.392120 31220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.392187 31220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:41.392236 31220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.401389 31220 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:40451
I20260925 04:33:41.401420 31350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:40451 every 8 connection(s)
I20260925 04:33:41.402257 31220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:41.403708 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31220
I20260925 04:33:41.403836 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:41.404182 27835 external_mini_cluster.cc:1428] Setting key 14c090618663c79e28771a9a8417808c
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:33:41.418283 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.404064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:42793 (local address 127.27.46.254:35363)
0925 04:33:41.404225 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:41.404230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:41.405216 (+ 986us) server_negotiation.cc:407] Connection header received
0925 04:33:41.407121 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:41.407124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:41.407169 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:41.407232 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:41.408552 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.409225 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.411087 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.411317 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.411398 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.414392 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:41.414398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:41.414404 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:41.414434 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:41.415870 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.416406 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.416413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.416417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.416501 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.416935 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.416937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.416939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.417127 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:41.417287 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:41.417914 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 04:33:41.418061 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:33:41.419348 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.404503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35363 (local address 127.27.46.194:42793)
0925 04:33:41.405046 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:41.405112 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:41.406894 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:41.407414 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:41.407425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:41.407903 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:41.408387 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.408401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.409366 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.409373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.410956 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.410972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.411499 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.411506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.411765 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.412399 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:41.412423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:41.414227 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:41.415969 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.415973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.415984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.416298 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.416601 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.416605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.416607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.416796 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.417236 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:41.417243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:41.417627 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 04:33:41.417830 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 04:33:41.418193 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260925 04:33:41.420523 31351 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35363
I20260925 04:33:41.420778 31351 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:41.421337 31351 heartbeater.cc:511] Master 127.27.46.254:35363 requested a full tablet report, sending...
I20260925 04:33:41.422528 31016 ts_manager.cc:194] Registered new tserver with Master: 80174a458bbd4f0793ea6ff83dd19d0b (127.27.46.194:40451)
I20260925 04:33:41.423110 31016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:42793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:41.434115 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:41.544356 31356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:41.544610 31356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:41.544651 31356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:41.548470 31356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:41.548516 31356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:41.548571 31356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:41.553308 31356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:41.554239 31356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:41.555212 31356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:41.564210 31362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:41.564219 31361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:41.564424 31356 server_base.cc:1034] running on GCE node
W20260925 04:33:41.564239 31364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:41.564834 31356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:41.565582 31356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:41.566780 31356 hybrid_clock.cc:648] HybridClock initialized: now 1790310821566758 us; error 37 us; skew 500 ppm
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:41.569591 31356 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:41.570384 31356 webserver.cc:460] Webserver started at http://127.27.46.195:46139/ using document root <none> and password file <none>
I20260925 04:33:41.570926 31356 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:41.570993 31356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:41.571162 31356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:41.573081 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "d3d83324c0d94c4fa41a041e2a258d78"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "97522e2ee057e012f5b9232b2585b671"
server_key_iv: "12c2753f6d158d17d0609804374912f8"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.573633 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "d3d83324c0d94c4fa41a041e2a258d78"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "97522e2ee057e012f5b9232b2585b671"
server_key_iv: "12c2753f6d158d17d0609804374912f8"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.576982 31356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 04:33:41.579295 31371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.580469 31356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:41.580579 31356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "d3d83324c0d94c4fa41a041e2a258d78"
format_stamp: "Formatted at 2026-09-25 04:33:41 on dist-test-slave-r8r3"
server_key: "97522e2ee057e012f5b9232b2585b671"
server_key_iv: "12c2753f6d158d17d0609804374912f8"
server_key_version: "encryptionkey@0"
I20260925 04:33:41.580698 31356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:41.609622 31356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:41.612633 31356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:41.612843 31356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:41.613511 31356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:41.614358 31356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:41.614430 31356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.614503 31356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:41.614560 31356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:41.623862 31356 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:35347
I20260925 04:33:41.623899 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:35347 every 8 connection(s)
I20260925 04:33:41.624764 31356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:41.630349 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31356
I20260925 04:33:41.630481 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:41.630805 27835 external_mini_cluster.cc:1428] Setting key bd780404ca7dca38df9309010faf9c5b
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:41.638568 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.626681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:38057 (local address 127.27.46.254:35363)
0925 04:33:41.626848 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:41.626853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:41.627718 (+ 865us) server_negotiation.cc:407] Connection header received
0925 04:33:41.628995 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:41.628998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:41.629045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:41.629126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:41.630346 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.631122 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.632461 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.632772 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.632904 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.635367 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:41.635370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:41.635373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:41.635404 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:41.636569 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.637060 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.637064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.637066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.637151 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.637482 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.637485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.637487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.637686 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:41.637795 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:41.638258 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 04:33:41.638398 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:33:41.639410 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.627108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35363 (local address 127.27.46.195:38057)
0925 04:33:41.627563 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:41.627606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:41.628808 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:41.629286 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:41.629294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:41.629725 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:41.630203 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.630217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.631254 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.631260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.632354 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:41.632363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.632941 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.632945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:41.633121 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.633760 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:41.633776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:41.635206 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:41.636694 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.636699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.636713 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.636956 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.637244 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:41.637247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:41.637249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:41.637385 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:41.637793 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:41.637800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:41.638035 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 04:33:41.638217 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:33:41.638539 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260925 04:33:41.640415 31494 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35363
I20260925 04:33:41.640645 31494 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:41.641101 31494 heartbeater.cc:511] Master 127.27.46.254:35363 requested a full tablet report, sending...
I20260925 04:33:41.642356 31016 ts_manager.cc:194] Registered new tserver with Master: d3d83324c0d94c4fa41a041e2a258d78 (127.27.46.195:35347)
I20260925 04:33:41.642855 31016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:38057
I20260925 04:33:41.644625 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[30968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310821, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:41.668773 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:41.660712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.27.46.254:35363)
0925 04:33:41.660867 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:41.660872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:41.661064 (+ 192us) server_negotiation.cc:407] Connection header received
0925 04:33:41.661228 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:41.661230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:41.661270 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:41.661348 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:41.662058 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.662537 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.663315 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:41.663519 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:41.663630 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:41.666043 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:41.666047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:41.666049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:41.666073 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:41.667193 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.667570 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.667573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.667575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.667651 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:41.667903 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:41.667906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:41.667908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:41.668111 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:41.668243 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:41.668477 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 04:33:41.668606 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 04:33:41.669168 31016 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57006
I20260925 04:33:41.670051 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31080
I20260925 04:33:41.676872 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31220
I20260925 04:33:41.682590 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31356
I20260925 04:33:41.688129 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 30982
2026-09-25T04:33:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2098 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[31506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[31506](info): set up 2 sockets
Sep 25 04:33:41 dist-test-slave-r8r3 krb5kdc[31506](info): commencing operation
krb5kdc: starting...
W20260925 04:33:42.745289 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:42 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:42.864537 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45065
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:42.974184 31519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:42.974442 31519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:42.974496 31519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:42.978458 31519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:42.978515 31519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:42.978538 31519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:42.978583 31519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:42.978621 31519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:42.984081 31519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:45065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:42.985126 31519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:42.986225 31519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:42.994906 31525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:42.995002 31527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:42.994906 31524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:42.995327 31519 server_base.cc:1034] running on GCE node
I20260925 04:33:42.995956 31519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:42.997114 31519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:42.998301 31519 hybrid_clock.cc:648] HybridClock initialized: now 1790310822998268 us; error 43 us; skew 500 ppm
Sep 25 04:33:42 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:43.001004 31519 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:43.002002 31519 webserver.cc:460] Webserver started at http://127.27.46.254:44943/ using document root <none> and password file <none>
I20260925 04:33:43.002573 31519 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:43.002662 31519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:43.002890 31519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:43.005002 31519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "d41846b62f214a869c3113fd1f4d93cf"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "9c03c0d177a70f1d9d1fe2073be062de"
server_key_iv: "1cf6cfad2ae44a291495343ad1e6c235"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.005661 31519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "d41846b62f214a869c3113fd1f4d93cf"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "9c03c0d177a70f1d9d1fe2073be062de"
server_key_iv: "1cf6cfad2ae44a291495343ad1e6c235"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.009327 31519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:43.011708 31534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.012866 31519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:43.013006 31519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "d41846b62f214a869c3113fd1f4d93cf"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "9c03c0d177a70f1d9d1fe2073be062de"
server_key_iv: "1cf6cfad2ae44a291495343ad1e6c235"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.013118 31519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:43.034639 31519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:43.037483 31519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:43.037752 31519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:43.045028 31519 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:45065
I20260925 04:33:43.045060 31589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:45065 every 8 connection(s)
I20260925 04:33:43.046036 31519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:43.048593 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:43.050837 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31519
I20260925 04:33:43.050951 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:43.051244 27835 external_mini_cluster.cc:1428] Setting key b629eafb5d8d2537b735c82d11ca48f4
I20260925 04:33:43.054674 31590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf: Bootstrap starting.
I20260925 04:33:43.056841 31590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:43.057598 31590 log.cc:824] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:43.059386 31590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf: No bootstrap required, opened a new log
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:43.062495 31590 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } }
I20260925 04:33:43.062734 31590 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:43.062815 31590 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41846b62f214a869c3113fd1f4d93cf, State: Initialized, Role: FOLLOWER
I20260925 04:33:43.063282 31590 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } }
I20260925 04:33:43.063441 31590 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:43.063566 31590 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:43.063699 31590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:43.064818 31590 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } }
I20260925 04:33:43.065173 31590 leader_election.cc:302] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41846b62f214a869c3113fd1f4d93cf; no voters:
I20260925 04:33:43.065531 31590 leader_election.cc:288] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:43.065667 31595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:43.065913 31595 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [term 1 LEADER]: Becoming Leader. State: Replica: d41846b62f214a869c3113fd1f4d93cf, State: Running, Role: LEADER
I20260925 04:33:43.066239 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.052666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.27.46.254:45065)
0925 04:33:43.053082 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.053097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.053130 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:33:43.053903 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.053920 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.054232 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.054563 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.055388 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.056387 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.057307 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.057882 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.058043 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.060558 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.060565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.060588 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.060616 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.063067 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.063556 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.063562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.063568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.063731 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.064028 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.064031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.064033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.064541 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.064738 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.064948 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.065241 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260925 04:33:43.066277 31595 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } }
I20260925 04:33:43.066792 31590 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:43.067557 31595 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41846b62f214a869c3113fd1f4d93cf. Latest consensus state: current_term: 1 leader_uuid: "d41846b62f214a869c3113fd1f4d93cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } } }
I20260925 04:33:43.068300 31595 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:43.068192 31597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41846b62f214a869c3113fd1f4d93cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41846b62f214a869c3113fd1f4d93cf" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45065 } } }
I20260925 04:33:43.068624 31597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:43.069785 31606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:43.072340 31606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:43.079437 31606 catalog_manager.cc:1347] Generated new cluster ID: 6074096d4b0a4e42b31469f82e5683a8
I20260925 04:33:43.079520 31606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:43.089126 31606 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:43.090354 31606 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:43.098702 31606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41846b62f214a869c3113fd1f4d93cf: Generated new TSK 0
I20260925 04:33:43.099368 31606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:43.134703 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:43.278527 31616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:43.278764 31616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:43.278806 31616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:43.282851 31616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:43.282907 31616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:43.282961 31616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:43.287914 31616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:43.288877 31616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:43.290118 31616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:43.300029 31624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:43.300197 31616 server_base.cc:1034] running on GCE node
W20260925 04:33:43.300053 31622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:43.300074 31621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:43.300722 31616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:43.301376 31616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:43.302585 31616 hybrid_clock.cc:648] HybridClock initialized: now 1790310823302573 us; error 24 us; skew 500 ppm
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:43.305261 31616 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:43.306131 31616 webserver.cc:460] Webserver started at http://127.27.46.193:43995/ using document root <none> and password file <none>
I20260925 04:33:43.306677 31616 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:43.306768 31616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:43.306994 31616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:43.308991 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "81bfb9733af2443b877eeffdcd486b9a"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "81bbf3b97d1305303a868503bf4d86e3"
server_key_iv: "ebe5dc27ce0ea12ca8453b28e1f5fd3d"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.309628 31616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "81bfb9733af2443b877eeffdcd486b9a"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "81bbf3b97d1305303a868503bf4d86e3"
server_key_iv: "ebe5dc27ce0ea12ca8453b28e1f5fd3d"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.313144 31616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:33:43.315562 31631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.316778 31616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:43.316934 31616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "81bfb9733af2443b877eeffdcd486b9a"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "81bbf3b97d1305303a868503bf4d86e3"
server_key_iv: "ebe5dc27ce0ea12ca8453b28e1f5fd3d"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.317057 31616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:43.334106 31616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:43.336516 31616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:43.336699 31616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:43.337246 31616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:43.338058 31616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:43.338102 31616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.338165 31616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:43.338194 31616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.346714 31616 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:36055
I20260925 04:33:43.346741 31747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:36055 every 8 connection(s)
I20260925 04:33:43.347539 31616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:43.351341 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31616
I20260925 04:33:43.351493 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:43.351925 27835 external_mini_cluster.cc:1428] Setting key ab91d99357392f1a10acaf299567acc9
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:43.362277 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.349281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:58955 (local address 127.27.46.254:45065)
0925 04:33:43.349427 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.349431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.350272 (+ 841us) server_negotiation.cc:407] Connection header received
0925 04:33:43.351723 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.351728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.351795 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.351903 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.353161 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.353786 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.354915 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.355137 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.355240 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.358123 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.358126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.358128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.358152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.360065 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.360571 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.360577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.360580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.360679 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.361011 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.361015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.361018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.361260 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.361384 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.361939 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.362094 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:33:43.363004 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.349630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45065 (local address 127.27.46.193:58955)
0925 04:33:43.350125 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:43.350166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:43.351481 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:43.352095 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:43.352103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:43.352521 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:43.353013 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.353026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.353933 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.353937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.354786 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.354795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.355291 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.355295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.355452 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.356065 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:43.356080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:43.357921 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:43.360201 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.360219 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.360230 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.360468 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.360780 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.360782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.360784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.360912 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.361383 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:43.361388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:43.361681 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 04:33:43.361846 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 04:33:43.362095 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":129,"threads_started":1}
I20260925 04:33:43.364368 31748 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45065
I20260925 04:33:43.364610 31748 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:43.365170 31748 heartbeater.cc:511] Master 127.27.46.254:45065 requested a full tablet report, sending...
I20260925 04:33:43.366896 31552 ts_manager.cc:194] Registered new tserver with Master: 81bfb9733af2443b877eeffdcd486b9a (127.27.46.193:36055)
I20260925 04:33:43.368280 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:58955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:43.382776 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:43.495630 31753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:43.496040 31753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:43.496104 31753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:43.500540 31753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:43.500595 31753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:43.500651 31753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:43.505864 31753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:43.506824 31753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:43.507802 31753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:43.517316 31759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:43.517326 31761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:43.517326 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:43.517694 31753 server_base.cc:1034] running on GCE node
I20260925 04:33:43.518096 31753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:43.518679 31753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:43.519827 31753 hybrid_clock.cc:648] HybridClock initialized: now 1790310823519809 us; error 31 us; skew 500 ppm
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:43.522655 31753 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:43.523442 31753 webserver.cc:460] Webserver started at http://127.27.46.194:43319/ using document root <none> and password file <none>
I20260925 04:33:43.523980 31753 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:43.524036 31753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:43.524194 31753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:43.526230 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "63786a6849334403b6d72d7ebc94bdc4"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "1619759a95288f319d44ab37239e6a2c"
server_key_iv: "8961eebdd90fd23c9f11cd324773852e"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.526690 31753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "63786a6849334403b6d72d7ebc94bdc4"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "1619759a95288f319d44ab37239e6a2c"
server_key_iv: "8961eebdd90fd23c9f11cd324773852e"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.530229 31753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:43.532586 31768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.533815 31753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:43.533964 31753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "63786a6849334403b6d72d7ebc94bdc4"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "1619759a95288f319d44ab37239e6a2c"
server_key_iv: "8961eebdd90fd23c9f11cd324773852e"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.534070 31753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:43.547840 31753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:43.550966 31753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:43.551157 31753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:43.551712 31753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:43.552474 31753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:43.552536 31753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.552596 31753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:43.552644 31753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.561120 31753 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:35685
I20260925 04:33:43.561151 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:35685 every 8 connection(s)
I20260925 04:33:43.561985 31753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:43.569152 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31753
I20260925 04:33:43.569278 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:43.569628 27835 external_mini_cluster.cc:1428] Setting key 3c335fb0bf02a51bb76e811d09b44006
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:43.575901 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.563666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:56177 (local address 127.27.46.254:45065)
0925 04:33:43.563797 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.563801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.564586 (+ 785us) server_negotiation.cc:407] Connection header received
0925 04:33:43.565937 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.565940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.565986 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.566056 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.567197 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.567698 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.569098 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.569415 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.569561 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.572110 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.572114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.572116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.572140 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.573639 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.574236 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.574239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.574240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.574313 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.574707 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.574709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.574710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.574889 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.575020 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.575673 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.575784 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:43.577175 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.564011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45065 (local address 127.27.46.194:56177)
0925 04:33:43.564435 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:43.564474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:43.565749 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:43.566209 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:43.566215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:43.566530 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:43.567046 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.567058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.567881 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.567887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.568916 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.568926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.569604 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.569609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.569832 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.570507 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:43.570523 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:43.571942 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:43.573748 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.573757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.573774 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.574134 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.574419 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.574425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.574430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.574617 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.575041 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:43.575050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:43.575400 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 04:33:43.575629 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 04:33:43.576001 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260925 04:33:43.578405 31884 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45065
I20260925 04:33:43.578645 31884 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:43.579099 31884 heartbeater.cc:511] Master 127.27.46.254:45065 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:33:43.580155 31552 ts_manager.cc:194] Registered new tserver with Master: 63786a6849334403b6d72d7ebc94bdc4 (127.27.46.194:35685)
I20260925 04:33:43.580868 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:56177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:43.596771 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:43.706082 31890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:43.706354 31890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:43.706427 31890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:43.710500 31890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:43.710572 31890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:43.710661 31890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:43.716168 31890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:43.717344 31890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:43.718392 31890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:43.728063 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:43.728055 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:43.728055 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:43.728312 31890 server_base.cc:1034] running on GCE node
I20260925 04:33:43.728736 31890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:43.729558 31890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:43.730780 31890 hybrid_clock.cc:648] HybridClock initialized: now 1790310823730763 us; error 30 us; skew 500 ppm
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:43.733793 31890 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:43.734669 31890 webserver.cc:460] Webserver started at http://127.27.46.195:35065/ using document root <none> and password file <none>
I20260925 04:33:43.735173 31890 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:43.735236 31890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:43.735410 31890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:43.737542 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "4609aae8b1464c6ab9b877d1ca62601d"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "3e4bac8f43143d63c8d8307327c1b5b0"
server_key_iv: "7932ee3f77648c8abc102a5abf8400d3"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.738200 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "4609aae8b1464c6ab9b877d1ca62601d"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "3e4bac8f43143d63c8d8307327c1b5b0"
server_key_iv: "7932ee3f77648c8abc102a5abf8400d3"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.742260 31890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:33:43.744697 31905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.746011 31890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 04:33:43.746120 31890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "4609aae8b1464c6ab9b877d1ca62601d"
format_stamp: "Formatted at 2026-09-25 04:33:43 on dist-test-slave-r8r3"
server_key: "3e4bac8f43143d63c8d8307327c1b5b0"
server_key_iv: "7932ee3f77648c8abc102a5abf8400d3"
server_key_version: "encryptionkey@0"
I20260925 04:33:43.746201 31890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:43.780149 31890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:43.783978 31890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:43.784152 31890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:43.784747 31890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:43.785660 31890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:43.785712 31890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.785758 31890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:43.785781 31890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:43.795658 31890 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:33291
I20260925 04:33:43.795682 32021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:33291 every 8 connection(s)
I20260925 04:33:43.796607 31890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:43.803107 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 31890
I20260925 04:33:43.803258 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:43.803582 27835 external_mini_cluster.cc:1428] Setting key 146186a5693e1749e2f21a590deb9f9a
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:43.811196 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.798543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36611 (local address 127.27.46.254:45065)
0925 04:33:43.798721 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.798727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.799694 (+ 967us) server_negotiation.cc:407] Connection header received
0925 04:33:43.801034 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.801037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.801099 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.801179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.802388 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.803026 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.804593 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.804912 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.805019 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.807529 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.807533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.807535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.807559 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.808991 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.809797 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.809800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.809801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.809890 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.810260 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.810263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.810264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.810430 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.810502 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.810971 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.811077 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:33:43.812014 32024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.799060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45065 (local address 127.27.46.195:36611)
0925 04:33:43.799538 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:43.799580 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:43.800811 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:43.801344 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:43.801351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:43.801783 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:43.802258 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.802271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.803191 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.803194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.804459 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:43.804474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.805089 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.805093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:43.805287 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.805937 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:43.805953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:43.807384 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:43.809278 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.809284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.809297 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.809698 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.810023 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:43.810026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:43.810027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:43.810167 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:43.810518 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:43.810523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:43.810760 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 04:33:43.810927 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 04:33:43.811170 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
I20260925 04:33:43.813287 32022 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45065
I20260925 04:33:43.813597 32022 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:43.814160 32022 heartbeater.cc:511] Master 127.27.46.254:45065 requested a full tablet report, sending...
I20260925 04:33:43.815205 31552 ts_manager.cc:194] Registered new tserver with Master: 4609aae8b1464c6ab9b877d1ca62601d (127.27.46.195:33291)
I20260925 04:33:43.815761 31552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:36611
I20260925 04:33:43.817605 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:43.843150 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.833784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.27.46.193:36055)
0925 04:33:43.834124 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.834139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.834166 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:33:43.834244 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.834258 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.834443 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.834596 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.835296 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.836054 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.837006 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.837375 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.837532 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.839935 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.839943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.839956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.839979 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.841333 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.841818 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.841823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.841825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.841900 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.842182 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.842186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.842188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.842419 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.842638 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.842822 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.842972 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[31506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:43.853485 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:43.846062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.27.46.254:45065)
0925 04:33:43.846236 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:43.846239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:43.846268 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:33:43.846384 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:43.846385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:43.846425 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:43.846489 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:43.847076 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.847570 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.848303 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:43.848505 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:43.848576 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:43.850838 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:43.850841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:43.850844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:43.850868 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:43.852081 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.852446 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.852449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.852450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.852524 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:43.852753 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:43.852756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:43.852757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:43.852972 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:43.853072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:43.853272 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 04:33:43.853360 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 04:33:43.853942 31552 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53708
I20260925 04:33:43.854547 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31616
I20260925 04:33:43.862634 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31753
I20260925 04:33:43.869371 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31890
I20260925 04:33:43.875597 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 31519
2026-09-25T04:33:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2188 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[32035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[32035](info): set up 2 sockets
Sep 25 04:33:43 dist-test-slave-r8r3 krb5kdc[32035](info): commencing operation
krb5kdc: starting...
W20260925 04:33:44.937381 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:45.061440 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41137
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:45.174530 32048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:45.174952 32048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:45.175024 32048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:45.179499 32048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:45.179571 32048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:45.179592 32048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:45.179610 32048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:45.179625 32048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:45.185081 32048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:45.186199 32048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:45.187171 32048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:33:45.195838 32048 server_base.cc:1034] running on GCE node
W20260925 04:33:45.195834 32053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.195815 32056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.195809 32054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:45.196772 32048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:45.197973 32048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:45.199151 32048 hybrid_clock.cc:648] HybridClock initialized: now 1790310825199136 us; error 27 us; skew 500 ppm
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:45.206969 32048 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:45.207791 32048 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:45.207856 32048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:45.208027 32048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:45.210144 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "e44be075049c4d51995ac6c05c91c9d7"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "1ba9747e0dfac43fbb526b186e8a80f7"
server_key_iv: "c7150cb06f2d0337d6a3d14a46551ab3"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.210660 32048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "e44be075049c4d51995ac6c05c91c9d7"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "1ba9747e0dfac43fbb526b186e8a80f7"
server_key_iv: "c7150cb06f2d0337d6a3d14a46551ab3"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.214489 32048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:33:45.216940 32062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.218263 32048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:45.218434 32048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "e44be075049c4d51995ac6c05c91c9d7"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "1ba9747e0dfac43fbb526b186e8a80f7"
server_key_iv: "c7150cb06f2d0337d6a3d14a46551ab3"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.218535 32048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:45.231621 32048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:45.235002 32048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:45.235216 32048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:45.242333 32048 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:41137
I20260925 04:33:45.242620 32115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:41137 every 8 connection(s)
I20260925 04:33:45.242964 32048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:45.245749 32116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:45.248039 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32048
I20260925 04:33:45.248199 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:45.248539 27835 external_mini_cluster.cc:1428] Setting key 31835e5427d0ee159178413244a0aadd
I20260925 04:33:45.252776 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: Bootstrap starting.
I20260925 04:33:45.255132 32116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:45.255882 32116 log.cc:824] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:45.257984 32116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: No bootstrap required, opened a new log
I20260925 04:33:45.261572 32116 raft_consensus.cc:348] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } }
I20260925 04:33:45.261888 32116 raft_consensus.cc:374] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:45.261965 32116 raft_consensus.cc:729] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44be075049c4d51995ac6c05c91c9d7, State: Initialized, Role: FOLLOWER
I20260925 04:33:45.262456 32116 consensus_queue.cc:260] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } }
I20260925 04:33:45.262586 32116 raft_consensus.cc:388] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:45.262672 32116 raft_consensus.cc:482] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:45.262760 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.250028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.27.46.254:41137)
0925 04:33:45.250506 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:45.250521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:45.250556 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:33:45.251210 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:45.251226 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:45.251544 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:45.251873 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:45.252693 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.253769 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.254626 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.255081 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.255249 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.257928 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:45.257933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:45.257945 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:45.257970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:45.259758 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.260227 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.260232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.260237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.260344 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.260648 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.260652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.260654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.261090 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:45.261301 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:45.261555 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 04:33:45.261853 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":155,"threads_started":1}
I20260925 04:33:45.262773 32116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:45.263823 32116 raft_consensus.cc:504] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } }
I20260925 04:33:45.264187 32116 leader_election.cc:302] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e44be075049c4d51995ac6c05c91c9d7; no voters:
I20260925 04:33:45.264529 32116 leader_election.cc:288] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:45.264637 32121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:45.264829 32121 raft_consensus.cc:686] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [term 1 LEADER]: Becoming Leader. State: Replica: e44be075049c4d51995ac6c05c91c9d7, State: Running, Role: LEADER
I20260925 04:33:45.265128 32121 consensus_queue.cc:237] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } }
I20260925 04:33:45.265900 32116 sys_catalog.cc:564] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:45.266860 32123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e44be075049c4d51995ac6c05c91c9d7. Latest consensus state: current_term: 1 leader_uuid: "e44be075049c4d51995ac6c05c91c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } } }
I20260925 04:33:45.266904 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e44be075049c4d51995ac6c05c91c9d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e44be075049c4d51995ac6c05c91c9d7" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41137 } } }
I20260925 04:33:45.266995 32123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:45.267030 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:45.268559 32130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:45.271574 32130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:45.277319 32130 catalog_manager.cc:1347] Generated new cluster ID: beaae983a6594983aa9302f021216c79
I20260925 04:33:45.277395 32130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:45.286048 32130 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:45.287019 32130 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:45.291332 32130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: Generated new TSK 0
I20260925 04:33:45.292024 32130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:45.321059 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:45.455300 32142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:45.455603 32142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:45.455651 32142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:45.459898 32142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:45.459965 32142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:45.460034 32142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:45.465427 32142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:45.466758 32142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:45.468005 32142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:45.478437 32148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.478412 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.478412 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:45.478786 32142 server_base.cc:1034] running on GCE node
I20260925 04:33:45.479274 32142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:45.479984 32142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:45.481200 32142 hybrid_clock.cc:648] HybridClock initialized: now 1790310825481186 us; error 25 us; skew 500 ppm
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:45.484117 32142 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:45.484855 32142 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:45.484925 32142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:45.485103 32142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:45.487335 32142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "e0927fc68c76bc26aaba710044345f61"
server_key_iv: "61032bd9b5593eb620abcf59929c3f70"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.487977 32142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "e0927fc68c76bc26aaba710044345f61"
server_key_iv: "61032bd9b5593eb620abcf59929c3f70"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.491811 32142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:33:45.494231 32158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.495535 32142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:45.495666 32142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "e0927fc68c76bc26aaba710044345f61"
server_key_iv: "61032bd9b5593eb620abcf59929c3f70"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.495782 32142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:45.515226 32142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:45.518532 32142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:45.518757 32142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:45.519429 32142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:45.520363 32142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:45.520419 32142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.520489 32142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:45.520520 32142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.530004 32142 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44039
I20260925 04:33:45.530032 32278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44039 every 8 connection(s)
I20260925 04:33:45.530665 32142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:45.538595 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32142
I20260925 04:33:45.538741 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:45.539143 27835 external_mini_cluster.cc:1428] Setting key cab855eca65c960c80905b2a6e1e754b
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:45.545187 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.532489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:46871 (local address 127.27.46.254:41137)
0925 04:33:45.532638 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:45.532645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:45.533653 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 04:33:45.534973 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:45.534976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:45.535033 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:45.535106 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:45.536337 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.536864 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.538103 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.538334 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.538416 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.541408 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:45.541412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:45.541414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:45.541437 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:45.542913 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.543434 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.543440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.543443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.543553 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.543908 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.543911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.543913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.544179 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:45.544291 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:45.544945 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 04:33:45.545060 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:45.546725 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.532962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41137 (local address 127.27.46.193:46871)
0925 04:33:45.533433 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:45.533478 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:45.534766 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:45.535280 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:45.535287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:45.535668 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:45.536193 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.536208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.536988 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.536991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.537993 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.538002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.538518 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.538523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.538740 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.539431 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:45.539452 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:45.541224 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:45.543031 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.543037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.543050 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.543323 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.543668 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.543670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.543672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.543812 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.544288 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:45.544293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:45.544666 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 04:33:45.544903 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 04:33:45.545417 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260925 04:33:45.548205 32279 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41137
I20260925 04:33:45.548475 32279 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:45.549002 32279 heartbeater.cc:511] Master 127.27.46.254:41137 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:33:45.550674 32079 ts_manager.cc:194] Registered new tserver with Master: b9f5fe9940f2433cba9f5a3e7aa07ede (127.27.46.193:44039)
I20260925 04:33:45.551991 32079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:46871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:45.570823 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:45.683985 32284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:45.684245 32284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:45.684331 32284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:45.688410 32284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:45.688472 32284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:45.688589 32284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:45.693935 32284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:45.695041 32284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:45.696144 32284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:45.705642 32290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.705667 32291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.705642 32293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:45.705894 32284 server_base.cc:1034] running on GCE node
I20260925 04:33:45.706619 32284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:45.708427 32284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:45.709576 32284 hybrid_clock.cc:648] HybridClock initialized: now 1790310825709533 us; error 50 us; skew 500 ppm
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:45.712216 32284 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:45.712996 32284 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:45.713088 32284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:45.713320 32284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:45.715415 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "0ff9859164784557b0834a2fb1e46522"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "2bcfac60a4edd0b1486216ada6c80769"
server_key_iv: "6f800771e082e4eaab9365573a407fa4"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.715994 32284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "0ff9859164784557b0834a2fb1e46522"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "2bcfac60a4edd0b1486216ada6c80769"
server_key_iv: "6f800771e082e4eaab9365573a407fa4"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.719756 32284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 04:33:45.722146 32299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.723487 32284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:33:45.723640 32284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "0ff9859164784557b0834a2fb1e46522"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "2bcfac60a4edd0b1486216ada6c80769"
server_key_iv: "6f800771e082e4eaab9365573a407fa4"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.723753 32284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:45.734467 32284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:45.737262 32284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:45.737457 32284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:45.738076 32284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:45.738842 32284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:45.738888 32284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.738929 32284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:45.738950 32284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.747467 32284 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:46661
I20260925 04:33:45.747490 32414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:46661 every 8 connection(s)
I20260925 04:33:45.748040 32284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:45.757196 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32284
I20260925 04:33:45.757330 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:45.757701 27835 external_mini_cluster.cc:1428] Setting key 01e5864a8ec7fa9b62483c878ce22d43
I20260925 04:33:45.761821 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.749890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:33193 (local address 127.27.46.254:41137)
0925 04:33:45.750034 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:45.750038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:45.750915 (+ 877us) server_negotiation.cc:407] Connection header received
0925 04:33:45.752219 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:45.752222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:45.752282 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:45.752389 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:45.753554 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.754110 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.755252 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.755489 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.755613 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.758130 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:45.758134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:45.758136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:45.758161 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:45.759652 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.760160 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.760166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.760171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.760264 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.760603 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.760609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.760613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.760864 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:45.760969 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:45.761480 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 04:33:45.761658 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:33:45.762727 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.750298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41137 (local address 127.27.46.194:33193)
0925 04:33:45.750766 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:45.750806 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:45.752004 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:45.752569 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:45.752576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:45.752920 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:45.753389 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.753402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.754244 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.754250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.755117 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.755126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.755658 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.755662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.755838 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.756369 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:45.756386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:45.757951 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:45.759778 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.759783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.759795 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.760055 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.760357 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.760359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.760361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.760501 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.760966 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:45.760972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:45.761232 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 04:33:45.761429 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 04:33:45.761849 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260925 04:33:45.764040 32415 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41137
I20260925 04:33:45.764289 32415 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:45.764746 32415 heartbeater.cc:511] Master 127.27.46.254:41137 requested a full tablet report, sending...
I20260925 04:33:45.765937 32079 ts_manager.cc:194] Registered new tserver with Master: 0ff9859164784557b0834a2fb1e46522 (127.27.46.194:46661)
I20260925 04:33:45.766825 32079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:33193
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:45.785663 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:45.898880 32420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:45.899189 32420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:45.899281 32420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:45.903545 32420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:45.903640 32420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:45.903736 32420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:45.909003 32420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:45.910144 32420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:45.911155 32420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:33:45.920656 32420 server_base.cc:1034] running on GCE node
W20260925 04:33:45.920712 32425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.920712 32428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:45.920820 32426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:45.921561 32420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:45.922219 32420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:45.923406 32420 hybrid_clock.cc:648] HybridClock initialized: now 1790310825923393 us; error 28 us; skew 500 ppm
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:45.926862 32420 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:45.927922 32420 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:45.928020 32420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:45.928236 32420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:45.930400 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "f566cb7f316c4665a8f1f50826a7ef39"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "ec76e0f62224c23bdce54f2adbe5d1d1"
server_key_iv: "734621e71919fc8a18d8d99319320186"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.930982 32420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "f566cb7f316c4665a8f1f50826a7ef39"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "ec76e0f62224c23bdce54f2adbe5d1d1"
server_key_iv: "734621e71919fc8a18d8d99319320186"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.934609 32420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:45.936919 32434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.938207 32420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:45.938336 32420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "f566cb7f316c4665a8f1f50826a7ef39"
format_stamp: "Formatted at 2026-09-25 04:33:45 on dist-test-slave-r8r3"
server_key: "ec76e0f62224c23bdce54f2adbe5d1d1"
server_key_iv: "734621e71919fc8a18d8d99319320186"
server_key_version: "encryptionkey@0"
I20260925 04:33:45.938444 32420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:45.964182 32420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:45.967697 32420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:45.967919 32420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:45.968540 32420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:45.969385 32420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:45.969458 32420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.969558 32420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:45.969604 32420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:45.978376 32420 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:39989
I20260925 04:33:45.978401 32550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:39989 every 8 connection(s)
I20260925 04:33:45.978976 32420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:45.982159 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32420
I20260925 04:33:45.982295 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:45.982616 27835 external_mini_cluster.cc:1428] Setting key c65ccadc080ee811f6cf6500f1cffbfb
Sep 25 04:33:45 dist-test-slave-r8r3 krb5kdc[32035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:45.991956 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.980711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:33321 (local address 127.27.46.254:41137)
0925 04:33:45.980891 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:45.980895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:45.981802 (+ 907us) server_negotiation.cc:407] Connection header received
0925 04:33:45.983261 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:45.983264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:45.983325 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:45.983429 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:45.984674 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.985219 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.986369 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.986583 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.986660 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.988996 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:45.988999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:45.989002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:45.989025 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:45.990265 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.990701 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.990703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.990705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.990775 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:45.991056 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:45.991059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:45.991061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:45.991210 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:45.991281 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:45.991752 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 04:33:45.991842 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:33:45.992908 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:45.981091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41137 (local address 127.27.46.195:33321)
0925 04:33:45.981648 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:45.981694 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:45.983041 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:45.983630 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:45.983637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:45.983987 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:45.984490 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.984510 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.985327 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.985329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.986265 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:45.986275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:45.986711 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:45.986714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:45.986848 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:45.987455 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:45.987471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:45.988851 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:45.990365 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.990369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.990380 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.990620 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.990852 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:45.990854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:45.990855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:45.990983 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:45.991294 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:45.991298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:45.991504 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 04:33:45.991708 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 04:33:45.991939 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260925 04:33:45.994218 32551 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41137
I20260925 04:33:45.994455 32551 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:45.994954 32551 heartbeater.cc:511] Master 127.27.46.254:41137 requested a full tablet report, sending...
I20260925 04:33:45.996189 32078 ts_manager.cc:194] Registered new tserver with Master: f566cb7f316c4665a8f1f50826a7ef39 (127.27.46.195:39989)
I20260925 04:33:45.996759 32078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:33321
I20260925 04:33:46.007983 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:33:46.016878 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.009980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.27.46.254:41137)
0925 04:33:46.010113 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.010116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.010351 (+ 235us) server_negotiation.cc:407] Connection header received
0925 04:33:46.010468 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.010470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.010508 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.010578 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:46.011251 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.011756 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.012512 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.012714 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.012789 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.014193 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:46.014197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:46.014198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:46.014217 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:46.015495 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:46.015873 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:46.015875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:46.015877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:46.015947 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:46.016170 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:46.016172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:46.016173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:46.016373 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:46.016473 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.016674 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.016764 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:33:46.019629 32078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:46.022814 32078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:46.037330 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.032175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.27.46.195:39989)
0925 04:33:46.032486 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.032497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.032520 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:33:46.032594 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.032600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.032793 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.032939 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:46.033266 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.034493 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.035734 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.036539 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.036655 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.036835 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.037058 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.037165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260925 04:33:46.037326 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.030923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39989 (local address 127.0.0.1:41068)
0925 04:33:46.031381 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.031434 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:46.031553 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:46.032988 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:46.032994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:46.033019 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:46.033148 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.033153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.034662 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.034665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.035610 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.035621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.036654 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.036659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.036808 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.036879 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 04:33:46.037010 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:33:46.037161 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":87,"threads_started":1}
I20260925 04:33:46.040848 32483 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 (DEFAULT_TABLE table=test-table [id=f673f13fac2e417ba8bf6d92583c3591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:46.041594 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.031827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.0.0.1:52370)
0925 04:33:46.032099 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.032114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:46.032204 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:46.036903 (+ 4699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:46.036907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:46.036927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:46.037071 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.037078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.038730 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.038737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.039900 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.039911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.041142 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.041145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.041247 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.041271 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:33:46.041343 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:33:46.041417 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":21,"thread_start_us":129,"threads_started":1}
I20260925 04:33:46.042160 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.031957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.27.46.193:44039)
0925 04:33:46.036116 (+ 4159us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.036124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.036155 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:33:46.036251 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.036257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.036515 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.036702 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:46.037221 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.038574 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.040043 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.040831 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.040939 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.041076 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.041888 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.042006 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4028,"thread_start_us":95,"threads_started":1}
I20260925 04:33:46.042213 32483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd3140ffeb4726a2fc093423d1b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:46.042450 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.036081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:46661 (local address 127.0.0.1:42254)
0925 04:33:46.036656 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.036672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:46.036754 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:46.037711 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:46.037713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:46.037728 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:46.037787 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.037790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.039136 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.039139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.040205 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.040212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.041965 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.041970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.042069 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.042104 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:33:46.042204 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:33:46.042297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260925 04:33:46.042852 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.036245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.27.46.194:46661)
0925 04:33:46.037064 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.037081 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.037102 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:33:46.037175 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.037181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.037371 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.037551 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:46.037909 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.038977 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.040330 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.041584 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.041736 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.041892 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.042416 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.042726 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":103,"threads_started":1}
I20260925 04:33:46.044993 32208 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 (DEFAULT_TABLE table=test-table [id=f673f13fac2e417ba8bf6d92583c3591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:46.046124 32208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd3140ffeb4726a2fc093423d1b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:46.045828 32348 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 (DEFAULT_TABLE table=test-table [id=f673f13fac2e417ba8bf6d92583c3591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:46.046826 32348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd3140ffeb4726a2fc093423d1b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:46.048599 32568 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Bootstrap starting.
I20260925 04:33:46.051023 32568 tablet_bootstrap.cc:654] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:46.052115 32568 log.cc:824] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:46.054160 32568 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: No bootstrap required, opened a new log
I20260925 04:33:46.054378 32568 ts_tablet_manager.cc:1397] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 04:33:46.055084 32570 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Bootstrap starting.
I20260925 04:33:46.057353 32570 tablet_bootstrap.cc:654] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:46.057812 32571 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Bootstrap starting.
I20260925 04:33:46.058670 32570 log.cc:824] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:46.058893 32568 raft_consensus.cc:348] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.059203 32568 raft_consensus.cc:374] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:46.059266 32568 raft_consensus.cc:729] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f566cb7f316c4665a8f1f50826a7ef39, State: Initialized, Role: FOLLOWER
I20260925 04:33:46.059806 32568 consensus_queue.cc:260] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.060031 32571 tablet_bootstrap.cc:654] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:46.060570 32551 heartbeater.cc:503] Master 127.27.46.254:41137 was elected leader, sending a full tablet report...
I20260925 04:33:46.060901 32568 ts_tablet_manager.cc:1428] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:33:46.060943 32571 log.cc:824] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:46.061226 32570 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: No bootstrap required, opened a new log
I20260925 04:33:46.061424 32570 ts_tablet_manager.cc:1397] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:33:46.063287 32571 tablet_bootstrap.cc:492] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: No bootstrap required, opened a new log
I20260925 04:33:46.063505 32571 ts_tablet_manager.cc:1397] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:33:46.064881 32570 raft_consensus.cc:348] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.065172 32570 raft_consensus.cc:374] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:46.065235 32570 raft_consensus.cc:729] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff9859164784557b0834a2fb1e46522, State: Initialized, Role: FOLLOWER
I20260925 04:33:46.065771 32570 consensus_queue.cc:260] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.066499 32415 heartbeater.cc:503] Master 127.27.46.254:41137 was elected leader, sending a full tablet report...
I20260925 04:33:46.066838 32570 ts_tablet_manager.cc:1428] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20260925 04:33:46.068275 32571 raft_consensus.cc:348] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.068570 32571 raft_consensus.cc:374] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:46.068636 32571 raft_consensus.cc:729] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f5fe9940f2433cba9f5a3e7aa07ede, State: Initialized, Role: FOLLOWER
I20260925 04:33:46.069116 32571 consensus_queue.cc:260] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.069859 32279 heartbeater.cc:503] Master 127.27.46.254:41137 was elected leader, sending a full tablet report...
I20260925 04:33:46.070222 32571 ts_tablet_manager.cc:1428] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260925 04:33:46.229858 32552 tablet.cc:2369] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:33:46.249049 32416 tablet.cc:2369] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:33:46.282091 32280 tablet.cc:2369] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:46.316905 32575 raft_consensus.cc:482] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:46.317178 32575 raft_consensus.cc:504] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.318365 32575 leader_election.cc:288] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f566cb7f316c4665a8f1f50826a7ef39 (127.27.46.195:39989), b9f5fe9940f2433cba9f5a3e7aa07ede (127.27.46.193:44039)
I20260925 04:33:46.322559 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.318750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:43873 (local address 127.27.46.195:39989)
0925 04:33:46.318889 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.318895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.319104 (+ 209us) server_negotiation.cc:407] Connection header received
0925 04:33:46.319212 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.319218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.319278 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.319358 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:46.319677 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.320240 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.321475 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.322008 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.322100 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.322141 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.322344 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.322419 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:33:46.322572 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.318612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39989 (local address 127.27.46.194:43873)
0925 04:33:46.319011 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.319024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:46.319124 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:46.319385 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:46.319389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:46.319408 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:46.319551 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.319573 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.320342 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.320346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.321336 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.321345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.322101 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.322105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.322201 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.322239 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:33:46.322312 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:33:46.322383 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":26,"thread_start_us":89,"threads_started":1}
I20260925 04:33:46.322886 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.318783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60687 (local address 127.27.46.193:44039)
0925 04:33:46.318925 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.318931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.319282 (+ 351us) server_negotiation.cc:407] Connection header received
0925 04:33:46.319416 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.319419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.319473 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.319555 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:46.319847 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.320580 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.321720 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.322183 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.322311 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.322357 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.322651 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.322752 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:46.322978 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.318632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.27.46.194:60687)
0925 04:33:46.319182 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.319195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:46.319293 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:46.319581 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:46.319583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:46.319598 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:46.319733 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.319741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.320696 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.320699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.321603 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:46.321613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.322324 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.322327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:46.322434 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.322457 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:33:46.322533 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:33:46.322603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":88,"thread_start_us":79,"threads_started":1}
I20260925 04:33:46.323323 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd3140ffeb4726a2fc093423d1b5e6" candidate_uuid: "0ff9859164784557b0834a2fb1e46522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f566cb7f316c4665a8f1f50826a7ef39" is_pre_election: true
I20260925 04:33:46.323798 32503 raft_consensus.cc:2454] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff9859164784557b0834a2fb1e46522 in term 0.
I20260925 04:33:46.323767 32230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd3140ffeb4726a2fc093423d1b5e6" candidate_uuid: "0ff9859164784557b0834a2fb1e46522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" is_pre_election: true
I20260925 04:33:46.324084 32230 raft_consensus.cc:2454] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ff9859164784557b0834a2fb1e46522 in term 0.
I20260925 04:33:46.324314 32300 leader_election.cc:302] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff9859164784557b0834a2fb1e46522, f566cb7f316c4665a8f1f50826a7ef39; no voters:
I20260925 04:33:46.324631 32575 raft_consensus.cc:2781] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:46.324721 32575 raft_consensus.cc:482] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:46.324764 32575 raft_consensus.cc:3037] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:46.337729 32575 raft_consensus.cc:504] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.338225 32575 leader_election.cc:288] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [CANDIDATE]: Term 1 election: Requested vote from peers f566cb7f316c4665a8f1f50826a7ef39 (127.27.46.195:39989), b9f5fe9940f2433cba9f5a3e7aa07ede (127.27.46.193:44039)
I20260925 04:33:46.338616 32230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd3140ffeb4726a2fc093423d1b5e6" candidate_uuid: "0ff9859164784557b0834a2fb1e46522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede"
I20260925 04:33:46.338629 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd3140ffeb4726a2fc093423d1b5e6" candidate_uuid: "0ff9859164784557b0834a2fb1e46522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f566cb7f316c4665a8f1f50826a7ef39"
I20260925 04:33:46.338766 32503 raft_consensus.cc:3037] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:46.338766 32230 raft_consensus.cc:3037] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:46.340001 32230 raft_consensus.cc:2454] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff9859164784557b0834a2fb1e46522 in term 1.
I20260925 04:33:46.340335 32303 leader_election.cc:302] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff9859164784557b0834a2fb1e46522, b9f5fe9940f2433cba9f5a3e7aa07ede; no voters:
I20260925 04:33:46.340413 32503 raft_consensus.cc:2454] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ff9859164784557b0834a2fb1e46522 in term 1.
I20260925 04:33:46.340545 32575 raft_consensus.cc:2781] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:46.340783 32575 raft_consensus.cc:686] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 1 LEADER]: Becoming Leader. State: Replica: 0ff9859164784557b0834a2fb1e46522, State: Running, Role: LEADER
I20260925 04:33:46.341153 32575 consensus_queue.cc:237] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } }
I20260925 04:33:46.344470 32078 catalog_manager.cc:5495] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ff9859164784557b0834a2fb1e46522 (127.27.46.194). New cstate: current_term: 1 leader_uuid: "0ff9859164784557b0834a2fb1e46522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ff9859164784557b0834a2fb1e46522" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:46.412907 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.409229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.27.46.194:46661)
0925 04:33:46.409400 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.409405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.409420 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:33:46.409581 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.409584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.409643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.409772 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:46.410063 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.410561 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.411426 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:46.411649 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:46.411760 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:46.412087 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:46.412500 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:46.412636 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.412697 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.412753 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:33:46.418761 32230 raft_consensus.cc:1261] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 1 FOLLOWER]: Refusing update from remote peer 0ff9859164784557b0834a2fb1e46522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:46.418759 32503 raft_consensus.cc:1261] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 1 FOLLOWER]: Refusing update from remote peer 0ff9859164784557b0834a2fb1e46522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:46.419497 32575 consensus_queue.cc:1035] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f566cb7f316c4665a8f1f50826a7ef39" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:46.419700 32579 consensus_queue.cc:1035] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f5fe9940f2433cba9f5a3e7aa07ede" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:46.428035 32584 mvcc.cc:205] Tried to move back new op lower bound from 7333113145004044288 to 7333113144696414208. Current Snapshot: MvccSnapshot[applied={T|T < 7333113145004044288}]
I20260925 04:33:46.429347 32583 mvcc.cc:205] Tried to move back new op lower bound from 7333113145004044288 to 7333113144696414208. Current Snapshot: MvccSnapshot[applied={T|T < 7333113145004044288}]
I20260925 04:33:46.436503 32078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:46.436699 32078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:46.439757 32078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e44be075049c4d51995ac6c05c91c9d7: Sending DeleteTablet for 3 replicas of tablet 45cd3140ffeb4726a2fc093423d1b5e6
I20260925 04:33:46.440506 32348 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:46 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:42254
I20260925 04:33:46.441045 32483 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:46 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260925 04:33:46.441525 32593 tablet_replica.cc:330] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: stopping tablet replica
I20260925 04:33:46.441875 32593 raft_consensus.cc:2229] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:46.442019 32208 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd3140ffeb4726a2fc093423d1b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:46 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:52370
I20260925 04:33:46.442405 32593 raft_consensus.cc:2258] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:46.443207 32594 tablet_replica.cc:330] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede: stopping tablet replica
I20260925 04:33:46.443230 32592 tablet_replica.cc:330] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: stopping tablet replica
I20260925 04:33:46.443493 32594 raft_consensus.cc:2229] T 45cd3140ffeb4726a2fc093423d1b5e6 P b9f5fe9940f2433cba9f5a3e7aa07ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:46.443493 32592 raft_consensus.cc:2229] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:46.443632 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32142
I20260925 04:33:46.444171 32592 raft_consensus.cc:2258] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:46.445648 32592 ts_tablet_manager.cc:1905] T 45cd3140ffeb4726a2fc093423d1b5e6 P 0ff9859164784557b0834a2fb1e46522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:46.446599 32593 ts_tablet_manager.cc:1905] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:33:46.451207 32066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:44039 (error 108)
I20260925 04:33:46.452049 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32284
I20260925 04:33:46.452288 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.451950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.0.0.1:52384)
0925 04:33:46.452101 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.452182 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:33:46.455396 32066 catalog_manager.cc:4584] TS b9f5fe9940f2433cba9f5a3e7aa07ede (127.27.46.193:44039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45cd3140ffeb4726a2fc093423d1b5e6: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44039: connect: Connection refused (error 111)
I20260925 04:33:46.508622 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.508252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.0.0.1:52396)
0925 04:33:46.508444 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.508517 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 04:33:46.509410 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.509184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.0.0.1:52408)
0925 04:33:46.509300 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.509353 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 04:33:46.539350 32593 ts_tablet_manager.cc:1918] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:46.539620 32593 log.cc:1196] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/45cd3140ffeb4726a2fc093423d1b5e6
I20260925 04:33:46.539965 32593 ts_tablet_manager.cc:1939] T 45cd3140ffeb4726a2fc093423d1b5e6 P f566cb7f316c4665a8f1f50826a7ef39: Deleting consensus metadata
I20260925 04:33:46.544746 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32420
I20260925 04:33:46.546679 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.544937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:46661 (local address 127.0.0.1:42272)
0925 04:33:46.546471 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.546544 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:46661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1417}
I20260925 04:33:46.569782 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32048
I20260925 04:33:46.569837 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.569432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44039 (local address 127.0.0.1:52424)
0925 04:33:46.569674 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:46.569734 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
2026-09-25T04:33:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2693 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T04:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:46Z Disabled control of system clock
I20260925 04:33:46.600682 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38643
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:33:46.719064 32599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:46.719321 32599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:46.719358 32599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:46.723488 32599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:46.723541 32599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:46.723557 32599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:46.723627 32599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:46.723658 32599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:46.728641 32599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:46.729810 32599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:46.731540 32599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:46.740098 32608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:46.740238 32605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:46.740109 32606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:46.740689 32599 server_base.cc:1034] running on GCE node
I20260925 04:33:46.741343 32599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:46.742830 32599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:46.744021 32599 hybrid_clock.cc:648] HybridClock initialized: now 1790310826743992 us; error 39 us; skew 500 ppm
I20260925 04:33:46.746009 32599 webserver.cc:460] Webserver started at http://127.27.46.254:34899/ using document root <none> and password file <none>
I20260925 04:33:46.746613 32599 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:46.746708 32599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:46.746938 32599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:46.748986 32599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "a7b13cdf1c06467c99103623f49c5dda"
format_stamp: "Formatted at 2026-09-25 04:33:46 on dist-test-slave-r8r3"
server_key: "2ce9b3693d1b1dc350b8f465c314ca72"
server_key_iv: "2c2df65070a20a3e5440e0059339fc18"
server_key_version: "encryptionkey@0"
I20260925 04:33:46.749627 32599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "a7b13cdf1c06467c99103623f49c5dda"
format_stamp: "Formatted at 2026-09-25 04:33:46 on dist-test-slave-r8r3"
server_key: "2ce9b3693d1b1dc350b8f465c314ca72"
server_key_iv: "2c2df65070a20a3e5440e0059339fc18"
server_key_version: "encryptionkey@0"
I20260925 04:33:46.753266 32599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:33:46.755693 32614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:46.756947 32599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:46.757112 32599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "a7b13cdf1c06467c99103623f49c5dda"
format_stamp: "Formatted at 2026-09-25 04:33:46 on dist-test-slave-r8r3"
server_key: "2ce9b3693d1b1dc350b8f465c314ca72"
server_key_iv: "2c2df65070a20a3e5440e0059339fc18"
server_key_version: "encryptionkey@0"
I20260925 04:33:46.757229 32599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:46.761713 32599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:46.762315 32599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:46.762507 32599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:46.770885 32599 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38643
I20260925 04:33:46.770895 32666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38643 every 8 connection(s)
I20260925 04:33:46.771952 32599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:46.776732 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32599
I20260925 04:33:46.776768 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:46.776883 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:46.777190 27835 external_mini_cluster.cc:1428] Setting key 06c39943173137e97a92de4fe93ee058
I20260925 04:33:46.783358 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:46.778734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.27.46.254:38643)
0925 04:33:46.779413 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:46.779429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:46.779469 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:33:46.780276 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:46.780297 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:46.780671 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:46.780991 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:46.781961 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:46.781965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:46.781978 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:46.781990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:46.782050 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:46.782068 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:46.782200 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:46.782314 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:33:46.782580 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":146,"threads_started":1}
I20260925 04:33:46.784776 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: Bootstrap starting.
I20260925 04:33:46.787192 32667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:46.788090 32667 log.cc:824] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:46.790192 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: No bootstrap required, opened a new log
I20260925 04:33:46.794121 32667 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } }
I20260925 04:33:46.794389 32667 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:46.794480 32667 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b13cdf1c06467c99103623f49c5dda, State: Initialized, Role: FOLLOWER
I20260925 04:33:46.794968 32667 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } }
I20260925 04:33:46.795137 32667 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:46.795213 32667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:46.795327 32667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:46.838112 32667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } }
I20260925 04:33:46.838745 32667 leader_election.cc:302] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b13cdf1c06467c99103623f49c5dda; no voters:
I20260925 04:33:46.839229 32667 leader_election.cc:288] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:46.839397 32672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:46.839658 32672 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [term 1 LEADER]: Becoming Leader. State: Replica: a7b13cdf1c06467c99103623f49c5dda, State: Running, Role: LEADER
I20260925 04:33:46.840034 32672 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } }
I20260925 04:33:46.840816 32667 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:46.841413 32673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b13cdf1c06467c99103623f49c5dda. Latest consensus state: current_term: 1 leader_uuid: "a7b13cdf1c06467c99103623f49c5dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } } }
I20260925 04:33:46.841706 32673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:46.841878 32674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b13cdf1c06467c99103623f49c5dda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b13cdf1c06467c99103623f49c5dda" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38643 } } }
I20260925 04:33:46.841982 32674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:46.842307 32681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:46.845687 32681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:46.851797 32681 catalog_manager.cc:1347] Generated new cluster ID: e951b1459de342d18e28a416f55d4d3a
I20260925 04:33:46.851884 32681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:46.882875 32681 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:46.883249 32681 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:46.889962 32681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: Generated new TSK 0
I20260925 04:33:46.890666 32681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:33:46.896850 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--builtin_ntp_servers=127.27.46.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:33:47.022262 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:47.022532 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:47.022585 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:47.026764 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:47.026829 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:47.026893 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:47.031834 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:47.032932 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:47.034358 32692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:47.044557 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.044755 32692 server_base.cc:1034] running on GCE node
W20260925 04:33:47.044523 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:47.044575 32697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.045297 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:47.046025 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:47.047209 32692 hybrid_clock.cc:648] HybridClock initialized: now 1790310827047174 us; error 45 us; skew 500 ppm
I20260925 04:33:47.049012 32692 webserver.cc:460] Webserver started at http://127.27.46.193:34309/ using document root <none> and password file <none>
I20260925 04:33:47.049649 32692 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:47.049739 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:47.049998 32692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:47.052183 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "276af04e764685c802986e0867bb8427"
server_key_iv: "aaf81c0b320cbcf17dc9852d64d03d45"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.052750 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "276af04e764685c802986e0867bb8427"
server_key_iv: "aaf81c0b320cbcf17dc9852d64d03d45"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.056650 32692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:33:47.059326 32706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.060825 32692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:47.060974 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "276af04e764685c802986e0867bb8427"
server_key_iv: "aaf81c0b320cbcf17dc9852d64d03d45"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.061141 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:47.066246 32692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:47.066908 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:47.067104 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:47.067689 32692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:47.068603 32692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:47.068673 32692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.068748 32692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:47.068792 32692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.078289 32692 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:40815
I20260925 04:33:47.078428 353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:40815 every 8 connection(s)
I20260925 04:33:47.079236 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:47.084489 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 32692
I20260925 04:33:47.084640 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:47.084919 27835 external_mini_cluster.cc:1428] Setting key 0d40da645c6cafe228b244224d91ae0d
I20260925 04:33:47.085850 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.080993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:50479 (local address 127.27.46.254:38643)
0925 04:33:47.081130 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.081134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.082457 (+ 1323us) server_negotiation.cc:407] Connection header received
0925 04:33:47.083874 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.083877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.083915 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.084001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.085342 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.085345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.085347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.085358 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.085379 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.085387 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.085468 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.085693 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.085738 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:33:47.087018 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.081381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38643 (local address 127.27.46.193:50479)
0925 04:33:47.082276 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.082320 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.083667 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.084163 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.084171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.084261 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.085062 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.085064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.085106 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.085177 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.085528 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.085540 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.085548 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.085735 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.085989 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":301,"threads_started":1}
I20260925 04:33:47.087574 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--builtin_ntp_servers=127.27.46.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:33:47.088353 354 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38643
I20260925 04:33:47.088629 354 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:47.089061 354 heartbeater.cc:511] Master 127.27.46.254:38643 requested a full tablet report, sending...
I20260925 04:33:47.090778 32631 ts_manager.cc:194] Registered new tserver with Master: 309ddf2448fc49cabd8b0fe3a10a5bb6 (127.27.46.193:40815)
W20260925 04:33:47.204943 357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:47.205219 357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:47.205289 357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:47.209587 357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:47.209651 357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:47.209708 357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:47.214591 357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:47.215659 357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:47.217015 357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:47.226466 365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:47.226450 362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:47.226450 363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.226918 357 server_base.cc:1034] running on GCE node
I20260925 04:33:47.227339 357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:47.227977 357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:47.229161 357 hybrid_clock.cc:648] HybridClock initialized: now 1790310827229145 us; error 29 us; skew 500 ppm
I20260925 04:33:47.231004 357 webserver.cc:460] Webserver started at http://127.27.46.194:34109/ using document root <none> and password file <none>
I20260925 04:33:47.231568 357 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:47.231685 357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:47.231918 357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:47.234287 357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "ee4cdc808fb64dd389c260c648798179"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "aa208fc87841fd17a34c711379c0249c"
server_key_iv: "8dbdfe2375c5f2c85c97b92c409d2e0b"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.234839 357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "ee4cdc808fb64dd389c260c648798179"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "aa208fc87841fd17a34c711379c0249c"
server_key_iv: "8dbdfe2375c5f2c85c97b92c409d2e0b"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.238541 357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:47.241428 371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.242877 357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:33:47.243031 357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "ee4cdc808fb64dd389c260c648798179"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "aa208fc87841fd17a34c711379c0249c"
server_key_iv: "8dbdfe2375c5f2c85c97b92c409d2e0b"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.243148 357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:47.248068 357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:47.248682 357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:47.248872 357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:47.249400 357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:47.250365 357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:47.250450 357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.250523 357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:47.250560 357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.259363 357 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:35315
I20260925 04:33:47.259433 487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:35315 every 8 connection(s)
I20260925 04:33:47.260308 357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:47.264817 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 357
I20260925 04:33:47.264955 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:47.265250 27835 external_mini_cluster.cc:1428] Setting key 800aa5e2526bd73d89665b3953ea0eb6
I20260925 04:33:47.266887 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.262335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:46079 (local address 127.27.46.254:38643)
0925 04:33:47.262478 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.262482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.263563 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 04:33:47.264954 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.264957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.265003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.265126 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.266415 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.266418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.266420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.266430 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.266452 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.266460 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.266537 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.266704 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.266747 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:33:47.267508 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--builtin_ntp_servers=127.27.46.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:33:47.267793 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.262763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38643 (local address 127.27.46.194:46079)
0925 04:33:47.263412 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.263451 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.264736 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.265298 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.265305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.265432 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.266129 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.266132 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.266185 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.266269 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.266552 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.266561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.266573 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.266753 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.267003 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20260925 04:33:47.268992 488 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38643
I20260925 04:33:47.269251 488 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:47.269743 488 heartbeater.cc:511] Master 127.27.46.254:38643 requested a full tablet report, sending...
I20260925 04:33:47.270869 32631 ts_manager.cc:194] Registered new tserver with Master: ee4cdc808fb64dd389c260c648798179 (127.27.46.194:35315)
W20260925 04:33:47.387064 491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:47.387310 491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:47.387380 491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:47.394958 491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:47.395016 491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:47.395123 491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:47.400434 491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:47.401549 491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:47.402992 491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:47.412933 496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.413107 491 server_base.cc:1034] running on GCE node
W20260925 04:33:47.412986 499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:47.413372 497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.413820 491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:47.414484 491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:47.415673 491 hybrid_clock.cc:648] HybridClock initialized: now 1790310827415658 us; error 25 us; skew 500 ppm
I20260925 04:33:47.417558 491 webserver.cc:460] Webserver started at http://127.27.46.195:43429/ using document root <none> and password file <none>
I20260925 04:33:47.418140 491 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:47.418233 491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:47.418463 491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:47.420550 491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "e3191bb093ca45c9acd594a832edd7e9"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "bd2f142937d6f2c9ea94867a497b6e00"
server_key_iv: "8ce79ea54fcab32fecf7e4a166f7d2d8"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.421124 491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "e3191bb093ca45c9acd594a832edd7e9"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "bd2f142937d6f2c9ea94867a497b6e00"
server_key_iv: "8ce79ea54fcab32fecf7e4a166f7d2d8"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.424966 491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:33:47.427636 505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.428854 491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:33:47.428982 491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "e3191bb093ca45c9acd594a832edd7e9"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "bd2f142937d6f2c9ea94867a497b6e00"
server_key_iv: "8ce79ea54fcab32fecf7e4a166f7d2d8"
server_key_version: "encryptionkey@0"
I20260925 04:33:47.429101 491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:47.433830 491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:47.434486 491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:47.434664 491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:47.435175 491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:47.436034 491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:47.436129 491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.436201 491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:47.436236 491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:47.445580 491 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:37761
I20260925 04:33:47.445631 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:37761 every 8 connection(s)
I20260925 04:33:47.446537 491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:47.452778 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.448350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52125 (local address 127.27.46.254:38643)
0925 04:33:47.448512 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.448518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.449682 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 04:33:47.451012 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.451016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.451062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.451144 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.452265 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.452268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.452271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.452284 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.452307 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.452323 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.452423 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.452588 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.452638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:47.453702 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.448831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38643 (local address 127.27.46.195:52125)
0925 04:33:47.449488 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.449585 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.450802 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.451303 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.451310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.451397 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.451986 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.451988 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.452028 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.452103 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.452414 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.452423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.452430 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.452614 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.452853 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260925 04:33:47.454612 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 491
I20260925 04:33:47.454749 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:47.454973 622 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38643
I20260925 04:33:47.455078 27835 external_mini_cluster.cc:1428] Setting key 97053e031dfcd8e3c0beac506351442a
I20260925 04:33:47.455214 622 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:47.455652 622 heartbeater.cc:511] Master 127.27.46.254:38643 requested a full tablet report, sending...
I20260925 04:33:47.456754 32628 ts_manager.cc:194] Registered new tserver with Master: e3191bb093ca45c9acd594a832edd7e9 (127.27.46.195:37761)
I20260925 04:33:47.469005 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:33:47.472610 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.471032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.27.46.254:38643)
0925 04:33:47.471198 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.471202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.471350 (+ 148us) server_negotiation.cc:407] Connection header received
0925 04:33:47.471445 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.471447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.471481 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.471548 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.472225 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.472227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.472229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.472240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.472258 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.472265 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.472334 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.472395 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.472430 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:33:47.475724 32628 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:47.479063 32628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:47.490955 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.487897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.27.46.193:40815)
0925 04:33:47.488458 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.488472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.488496 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:33:47.488565 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.488570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.488755 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.488902 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.490282 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.490288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.490297 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.490306 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.490337 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.490360 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.490455 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.490758 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.490815 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":91,"threads_started":1}
I20260925 04:33:47.490942 636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.487232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:37761 (local address 127.0.0.1:44426)
0925 04:33:47.488028 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.488090 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.488197 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.489061 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.489064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.489081 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.490053 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.490055 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.490064 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.490086 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.490439 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.490453 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.490462 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.490588 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.490661 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"spinlock_wait_cycles":9728,"thread_start_us":74,"threads_started":1}
I20260925 04:33:47.491139 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.487360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.27.46.195:37761)
0925 04:33:47.488697 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.488707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.488727 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:33:47.488774 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.488780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.488928 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.489276 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.490218 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.490225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.490239 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.490255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.490300 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.490337 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.490455 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.490631 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.490707 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":144,"threads_started":1}
I20260925 04:33:47.491772 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.487611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:40815 (local address 127.0.0.1:46908)
0925 04:33:47.488387 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.488400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.488466 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.488908 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.488915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.488991 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.490017 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.490019 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.490030 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.490056 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.490500 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.490503 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.490506 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.490610 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.490672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":8,"thread_start_us":98,"threads_started":1}
I20260925 04:33:47.492748 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.488020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.27.46.194:35315)
0925 04:33:47.488427 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.488441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.488464 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:33:47.488519 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.488524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.488740 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.489276 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.491739 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.491748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.491762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.491778 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.491824 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.491857 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.492411 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.492463 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.492545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":112,"threads_started":1}
I20260925 04:33:47.492297 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.487326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35315 (local address 127.0.0.1:51110)
0925 04:33:47.488027 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.488078 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.488196 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.488909 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.488915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.489005 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.489793 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.489797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.489821 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.489900 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.491966 (+ 2066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.491986 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.491992 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.492066 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.492105 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":12288,"thread_start_us":107,"threads_started":1}
I20260925 04:33:47.494242 553 tablet_service.cc:1467] Processing CreateTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd (DEFAULT_TABLE table=test-table [id=7d27cb67216541a7b7136e4e95e8195e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:47.495414 553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c8535b17ed41359b84f2ac4e7a14fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:47.495720 32754 tablet_service.cc:1467] Processing CreateTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd (DEFAULT_TABLE table=test-table [id=7d27cb67216541a7b7136e4e95e8195e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:47.495954 419 tablet_service.cc:1467] Processing CreateTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd (DEFAULT_TABLE table=test-table [id=7d27cb67216541a7b7136e4e95e8195e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:47.496754 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c8535b17ed41359b84f2ac4e7a14fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:47.496985 419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c8535b17ed41359b84f2ac4e7a14fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:47.502638 639 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Bootstrap starting.
I20260925 04:33:47.503911 640 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Bootstrap starting.
I20260925 04:33:47.504030 641 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Bootstrap starting.
I20260925 04:33:47.505048 639 tablet_bootstrap.cc:654] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:47.505903 640 tablet_bootstrap.cc:654] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:47.505949 639 log.cc:824] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:47.506090 641 tablet_bootstrap.cc:654] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:47.506737 640 log.cc:824] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:47.506815 641 log.cc:824] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:47.508162 639 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: No bootstrap required, opened a new log
I20260925 04:33:47.508368 639 ts_tablet_manager.cc:1397] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:33:47.508697 640 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: No bootstrap required, opened a new log
I20260925 04:33:47.508864 640 ts_tablet_manager.cc:1397] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:33:47.509379 641 tablet_bootstrap.cc:492] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: No bootstrap required, opened a new log
I20260925 04:33:47.509588 641 ts_tablet_manager.cc:1397] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:33:47.513054 641 raft_consensus.cc:348] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.513303 641 raft_consensus.cc:374] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:47.513357 641 raft_consensus.cc:729] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309ddf2448fc49cabd8b0fe3a10a5bb6, State: Initialized, Role: FOLLOWER
I20260925 04:33:47.513226 640 raft_consensus.cc:348] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.513439 640 raft_consensus.cc:374] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:47.513485 640 raft_consensus.cc:729] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3191bb093ca45c9acd594a832edd7e9, State: Initialized, Role: FOLLOWER
I20260925 04:33:47.513600 639 raft_consensus.cc:348] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.513823 639 raft_consensus.cc:374] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:47.513875 639 raft_consensus.cc:729] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4cdc808fb64dd389c260c648798179, State: Initialized, Role: FOLLOWER
I20260925 04:33:47.513909 641 consensus_queue.cc:260] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.513931 640 consensus_queue.cc:260] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.514300 639 consensus_queue.cc:260] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.514664 622 heartbeater.cc:503] Master 127.27.46.254:38643 was elected leader, sending a full tablet report...
I20260925 04:33:47.514923 641 ts_tablet_manager.cc:1428] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:33:47.514925 640 ts_tablet_manager.cc:1428] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 04:33:47.515115 639 ts_tablet_manager.cc:1428] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260925 04:33:47.515228 354 heartbeater.cc:503] Master 127.27.46.254:38643 was elected leader, sending a full tablet report...
I20260925 04:33:47.515407 488 heartbeater.cc:503] Master 127.27.46.254:38643 was elected leader, sending a full tablet report...
W20260925 04:33:47.580456 355 tablet.cc:2369] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:47.630532 645 raft_consensus.cc:482] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:47.630759 645 raft_consensus.cc:504] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.631928 645 leader_election.cc:288] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee4cdc808fb64dd389c260c648798179 (127.27.46.194:35315), 309ddf2448fc49cabd8b0fe3a10a5bb6 (127.27.46.193:40815)
I20260925 04:33:47.633919 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.632269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:41115 (local address 127.27.46.194:35315)
0925 04:33:47.632394 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.632399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.632417 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:33:47.632601 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.632605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.632656 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.632775 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.633482 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.633486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.633488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.633528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.633551 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.633562 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.633657 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.633746 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.633786 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:33:47.633919 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.632198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35315 (local address 127.27.46.195:41115)
0925 04:33:47.632336 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.632353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.632469 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.632798 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.632802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.632820 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.633327 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.633329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.633338 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.633364 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.633658 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.633662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.633667 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.633747 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.633786 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:33:47.634409 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:40815 (local address 127.27.46.195:60713)
0925 04:33:47.632843 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.632858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:47.632951 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:47.633258 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:47.633262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:47.633280 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:47.633780 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:33:47.633785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:47.633800 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:33:47.633829 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:47.634146 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:47.634149 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:47.634153 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:33:47.634229 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 04:33:47.634264 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":178,"thread_start_us":169,"threads_started":1}
I20260925 04:33:47.634426 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.632324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:60713 (local address 127.27.46.193:40815)
0925 04:33:47.632473 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.632478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.632945 (+ 467us) server_negotiation.cc:407] Connection header received
0925 04:33:47.633085 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.633093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.633127 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.633218 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.633953 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.633956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.633958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.633972 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.633995 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.634007 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.634106 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.634262 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.634309 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:33:47.634627 441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c8535b17ed41359b84f2ac4e7a14fd" candidate_uuid: "e3191bb093ca45c9acd594a832edd7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4cdc808fb64dd389c260c648798179" is_pre_election: true
I20260925 04:33:47.634917 441 raft_consensus.cc:2454] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3191bb093ca45c9acd594a832edd7e9 in term 0.
I20260925 04:33:47.634985 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c8535b17ed41359b84f2ac4e7a14fd" candidate_uuid: "e3191bb093ca45c9acd594a832edd7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" is_pre_election: true
I20260925 04:33:47.635239 307 raft_consensus.cc:2454] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3191bb093ca45c9acd594a832edd7e9 in term 0.
I20260925 04:33:47.635370 506 leader_election.cc:302] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3191bb093ca45c9acd594a832edd7e9, ee4cdc808fb64dd389c260c648798179; no voters:
I20260925 04:33:47.635649 645 raft_consensus.cc:2781] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:47.635733 645 raft_consensus.cc:482] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:47.635785 645 raft_consensus.cc:3037] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:47.693181 645 raft_consensus.cc:504] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
I20260925 04:33:47.693792 645 leader_election.cc:288] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [CANDIDATE]: Term 1 election: Requested vote from peers ee4cdc808fb64dd389c260c648798179 (127.27.46.194:35315), 309ddf2448fc49cabd8b0fe3a10a5bb6 (127.27.46.193:40815)
I20260925 04:33:47.694300 441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c8535b17ed41359b84f2ac4e7a14fd" candidate_uuid: "e3191bb093ca45c9acd594a832edd7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4cdc808fb64dd389c260c648798179"
I20260925 04:33:47.694331 307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63c8535b17ed41359b84f2ac4e7a14fd" candidate_uuid: "e3191bb093ca45c9acd594a832edd7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6"
I20260925 04:33:47.694468 441 raft_consensus.cc:3037] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:47.694469 307 raft_consensus.cc:3037] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:47.695987 307 raft_consensus.cc:2454] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3191bb093ca45c9acd594a832edd7e9 in term 1.
I20260925 04:33:47.695987 441 raft_consensus.cc:2454] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3191bb093ca45c9acd594a832edd7e9 in term 1.
I20260925 04:33:47.696411 507 leader_election.cc:302] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309ddf2448fc49cabd8b0fe3a10a5bb6, e3191bb093ca45c9acd594a832edd7e9; no voters:
I20260925 04:33:47.696630 645 raft_consensus.cc:2781] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:47.696874 645 raft_consensus.cc:686] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 1 LEADER]: Becoming Leader. State: Replica: e3191bb093ca45c9acd594a832edd7e9, State: Running, Role: LEADER
I20260925 04:33:47.697293 645 consensus_queue.cc:237] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } }
W20260925 04:33:47.697522 623 tablet.cc:2369] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:47.700899 32628 catalog_manager.cc:5495] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3191bb093ca45c9acd594a832edd7e9 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "e3191bb093ca45c9acd594a832edd7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3191bb093ca45c9acd594a832edd7e9" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:47.727424 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.725814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.27.46.195:37761)
0925 04:33:47.725966 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:47.725971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:47.725987 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:33:47.726146 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:47.726149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:47.726208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:47.726308 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:47.726954 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:47.726957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:47.726960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:47.726972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:47.726991 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:47.727006 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:47.727116 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:47.727230 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:33:47.727271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:33:47.733805 441 raft_consensus.cc:1261] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 1 FOLLOWER]: Refusing update from remote peer e3191bb093ca45c9acd594a832edd7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:47.733805 307 raft_consensus.cc:1261] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 1 FOLLOWER]: Refusing update from remote peer e3191bb093ca45c9acd594a832edd7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:47.734601 645 consensus_queue.cc:1035] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4cdc808fb64dd389c260c648798179" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:47.734797 649 consensus_queue.cc:1035] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309ddf2448fc49cabd8b0fe3a10a5bb6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:47.743429 653 mvcc.cc:205] Tried to move back new op lower bound from 7333113150389813248 to 7333113150251577344. Current Snapshot: MvccSnapshot[applied={T|T < 7333113150389813248 or (T in {7333113150389813248})}]
I20260925 04:33:47.750430 32631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:47.750617 32631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:47.753325 32631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b13cdf1c06467c99103623f49c5dda: Sending DeleteTablet for 3 replicas of tablet 63c8535b17ed41359b84f2ac4e7a14fd
I20260925 04:33:47.754201 419 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:47 UTC) from {username='slave'} at 127.0.0.1:51110
I20260925 04:33:47.754201 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:47 UTC) from {username='slave'} at 127.0.0.1:46908
I20260925 04:33:47.754705 661 tablet_replica.cc:330] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6: stopping tablet replica
I20260925 04:33:47.754911 553 tablet_service.cc:1514] Processing DeleteTablet for tablet 63c8535b17ed41359b84f2ac4e7a14fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:47 UTC) from {username='slave'} at 127.0.0.1:44426
I20260925 04:33:47.755025 661 raft_consensus.cc:2229] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:47.755352 661 raft_consensus.cc:2258] T 63c8535b17ed41359b84f2ac4e7a14fd P 309ddf2448fc49cabd8b0fe3a10a5bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:47.755365 663 tablet_replica.cc:330] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: stopping tablet replica
I20260925 04:33:47.755728 663 raft_consensus.cc:2229] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:47.755914 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32692
I20260925 04:33:47.756212 663 raft_consensus.cc:2258] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:47.756614 662 tablet_replica.cc:330] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: stopping tablet replica
I20260925 04:33:47.756888 662 raft_consensus.cc:2229] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:47.757169 662 raft_consensus.cc:2258] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:47.758042 663 ts_tablet_manager.cc:1905] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:47.758703 662 ts_tablet_manager.cc:1905] T 63c8535b17ed41359b84f2ac4e7a14fd P ee4cdc808fb64dd389c260c648798179: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:33:47.765859 32616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:40815 (error 108)
I20260925 04:33:47.766908 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.766617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:40815 (local address 127.0.0.1:46924)
0925 04:33:47.766724 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.766804 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:40815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 04:33:47.766953 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 357
W20260925 04:33:47.768258 32616 catalog_manager.cc:4584] TS 309ddf2448fc49cabd8b0fe3a10a5bb6 (127.27.46.193:40815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63c8535b17ed41359b84f2ac4e7a14fd: Network error: Client connection negotiation failed: client connection to 127.27.46.193:40815: connect: Connection refused (error 111)
I20260925 04:33:47.792721 663 ts_tablet_manager.cc:1918] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:47.792858 663 log.cc:1196] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/63c8535b17ed41359b84f2ac4e7a14fd
I20260925 04:33:47.793242 663 ts_tablet_manager.cc:1939] T 63c8535b17ed41359b84f2ac4e7a14fd P e3191bb093ca45c9acd594a832edd7e9: Deleting consensus metadata
I20260925 04:33:47.796051 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 491
I20260925 04:33:47.798136 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.797735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35315 (local address 127.0.0.1:51118)
0925 04:33:47.797951 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.798010 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:35315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 04:33:47.822261 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.821783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:40815 (local address 127.0.0.1:46926)
0925 04:33:47.822060 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.822141 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:40815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260925 04:33:47.823136 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.822845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:40815 (local address 127.0.0.1:46942)
0925 04:33:47.822986 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.823050 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:40815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:33:47.827342 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 32599
I20260925 04:33:47.827597 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:47.827376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:37761 (local address 127.0.0.1:44444)
0925 04:33:47.827491 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:47.827532 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.195:37761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
2026-09-25T04:33:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1258 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 04:33:47.850546 27835 mini_oidc.cc:152] Starting JWKS server
I20260925 04:33:47.852149 27835 webserver.cc:460] Webserver started at https://127.0.0.1:40609/ using document root <none> and password file <none>
I20260925 04:33:47.852290 27835 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:33:47.852510 27835 webserver.cc:460] Webserver started at http://127.0.0.1:39671/ using document root <none> and password file <none>
2026-09-25T04:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:47Z Disabled control of system clock
I20260925 04:33:47.861217 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35173
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35173
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40609/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/testchainca.cert with env {}
W20260925 04:33:47.971421 670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:47.971707 670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:47.971788 670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:47.975903 670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:47.975983 670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:47.976014 670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:47.976058 670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:47.976115 670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:33:47.976161 670 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:33:47.976197 670 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40609/jwks.json
I20260925 04:33:47.981547 670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40609/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:47.982591 670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:47.984149 670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:47.992451 675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.992609 670 server_base.cc:1034] running on GCE node
W20260925 04:33:47.992460 679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:47.992461 676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:47.993353 670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:47.994419 670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:47.995610 670 hybrid_clock.cc:648] HybridClock initialized: now 1790310827995581 us; error 39 us; skew 500 ppm
I20260925 04:33:47.997485 670 webserver.cc:460] Webserver started at http://127.27.46.254:45603/ using document root <none> and password file <none>
I20260925 04:33:47.998081 670 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:47.998175 670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:47.998370 670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:48.000319 670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "603a9fd566534d079b4f81cf8740f540"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "550f2399e22a748a4ecb9ac573a7c70f"
server_key_iv: "0b3c94a5ecd68924cff325458e34a6fc"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.000876 670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "603a9fd566534d079b4f81cf8740f540"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "550f2399e22a748a4ecb9ac573a7c70f"
server_key_iv: "0b3c94a5ecd68924cff325458e34a6fc"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.004961 670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 04:33:48.007843 685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:48.009106 670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:48.009258 670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "603a9fd566534d079b4f81cf8740f540"
format_stamp: "Formatted at 2026-09-25 04:33:47 on dist-test-slave-r8r3"
server_key: "550f2399e22a748a4ecb9ac573a7c70f"
server_key_iv: "0b3c94a5ecd68924cff325458e34a6fc"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.009374 670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:48.027938 670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:48.028679 670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:48.028915 670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:48.036407 670 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35173
I20260925 04:33:48.036432 739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35173 every 8 connection(s)
I20260925 04:33:48.037333 670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:48.040458 740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:48.045933 740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540: Bootstrap starting.
I20260925 04:33:48.046554 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 670
I20260925 04:33:48.046677 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:48.047012 27835 external_mini_cluster.cc:1428] Setting key 7f2509b3c8005ea064e1b0ef598ded25
I20260925 04:33:48.048957 740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:48.049762 740 log.cc:824] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:48.051532 740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540: No bootstrap required, opened a new log
I20260925 04:33:48.055140 740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } }
I20260925 04:33:48.055372 740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:48.055425 740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603a9fd566534d079b4f81cf8740f540, State: Initialized, Role: FOLLOWER
I20260925 04:33:48.055862 740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } }
I20260925 04:33:48.056011 740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:48.056098 740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:48.056221 740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:48.056375 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.048335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.27.46.254:35173)
0925 04:33:48.049109 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.049129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.049174 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:33:48.049836 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.049855 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.050276 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.050620 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:48.051031 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.052024 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.053013 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.053484 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.053721 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.054589 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:48.054594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:48.054606 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:48.054978 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:48.055029 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:48.055049 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:48.055159 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:48.055274 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 04:33:48.055541 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":126,"threads_started":1}
I20260925 04:33:48.057288 740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } }
I20260925 04:33:48.057672 740 leader_election.cc:302] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603a9fd566534d079b4f81cf8740f540; no voters:
I20260925 04:33:48.057941 740 leader_election.cc:288] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:48.058005 745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:48.058184 745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [term 1 LEADER]: Becoming Leader. State: Replica: 603a9fd566534d079b4f81cf8740f540, State: Running, Role: LEADER
I20260925 04:33:48.058468 745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } }
I20260925 04:33:48.059126 740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:48.059437 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:33:48.059615 746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603a9fd566534d079b4f81cf8740f540. Latest consensus state: current_term: 1 leader_uuid: "603a9fd566534d079b4f81cf8740f540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } } }
I20260925 04:33:48.059736 746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.060432 747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603a9fd566534d079b4f81cf8740f540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603a9fd566534d079b4f81cf8740f540" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35173 } } }
I20260925 04:33:48.060523 747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.061144 749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:48.063711 749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:48.068573 749 catalog_manager.cc:1347] Generated new cluster ID: 9db8d4e55c0347b9afb1527458a90797
I20260925 04:33:48.068636 749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:48.080888 749 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:48.082197 749 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:48.090494 749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603a9fd566534d079b4f81cf8740f540: Generated new TSK 0
I20260925 04:33:48.091109 749 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:33:48.125474 771 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 04:33:48.125638 771 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:33:48.126284 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.108630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.27.46.254:35173)
0925 04:33:48.108762 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.108767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.108963 (+ 196us) server_negotiation.cc:407] Connection header received
0925 04:33:48.109137 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.109140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.109195 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.109308 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:33:48.109563 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.110396 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.111213 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.111548 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.111652 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.111949 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:33:48.125877 (+ 13928us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:33:48.126128 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 04:33:48.126362 743 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 04:33:48.127444 770 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:48.108503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35173 (local address 127.0.0.1:38814)
0925 04:33:48.108862 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:48.108892 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:48.108994 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:48.109325 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:48.109327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:48.109333 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:33:48.109436 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.109441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.110510 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.110513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.111107 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.111114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.111661 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.111664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.111729 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.111770 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:33:48.126099 (+ 14329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:33:48.126312 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.46.254:35173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.46.254:35173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40609/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 04:33:48.128316 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 670
2026-09-25T04:33:48Z chronyd exiting
I20260925 04:33:48.166718 27835 test_util.cc:175] -----------------------------------------------
I20260925 04:33:48.166821 27835 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790310801918616-27835-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 04:33:48.167958 27835 mini_oidc.cc:152] Starting JWKS server
I20260925 04:33:48.168834 27835 webserver.cc:460] Webserver started at https://127.0.0.1:38393/ using document root <none> and password file <none>
I20260925 04:33:48.168964 27835 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:33:48.169231 27835 webserver.cc:460] Webserver started at http://127.0.0.1:40565/ using document root <none> and password file <none>
2026-09-25T04:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:48Z Disabled control of system clock
I20260925 04:33:48.178155 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33465
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:33465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/testchainca.cert with env {}
W20260925 04:33:48.296164 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:48.296442 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:48.296482 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:48.300555 777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:48.300621 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:48.300637 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:48.300653 777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:48.300668 777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:33:48.300684 777 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:33:48.300704 777 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38393/jwks.json
I20260925 04:33:48.306216 777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:33465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:48.307276 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:48.308262 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:48.316773 782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:48.316841 777 server_base.cc:1034] running on GCE node
W20260925 04:33:48.316789 785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:48.316867 783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:48.317626 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:48.318692 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:48.319846 777 hybrid_clock.cc:648] HybridClock initialized: now 1790310828319816 us; error 43 us; skew 500 ppm
I20260925 04:33:48.321774 777 webserver.cc:460] Webserver started at http://127.27.46.254:41893/ using document root <none> and password file <none>
I20260925 04:33:48.322304 777 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:48.322361 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:48.322521 777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:48.324501 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "b9305df0db664db08b45093bebaa3075"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "274e05294cec4494ff5a021ae17ffcc6"
server_key_iv: "af01726a0b663fb43c984f931de74202"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.325023 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "b9305df0db664db08b45093bebaa3075"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "274e05294cec4494ff5a021ae17ffcc6"
server_key_iv: "af01726a0b663fb43c984f931de74202"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.328487 777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:48.331032 792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:48.332231 777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:48.332329 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "b9305df0db664db08b45093bebaa3075"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "274e05294cec4494ff5a021ae17ffcc6"
server_key_iv: "af01726a0b663fb43c984f931de74202"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.332412 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:48.347549 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:48.348186 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:48.348336 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:48.355305 777 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:33465
I20260925 04:33:48.355312 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:33465 every 8 connection(s)
I20260925 04:33:48.356230 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:48.359370 846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:48.363554 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 777
I20260925 04:33:48.363677 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:48.363960 27835 external_mini_cluster.cc:1428] Setting key 0d642f0366c66ebed5702830cb55d6ec
I20260925 04:33:48.365041 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075: Bootstrap starting.
I20260925 04:33:48.367360 846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:48.368112 846 log.cc:824] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:48.370198 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075: No bootstrap required, opened a new log
I20260925 04:33:48.374001 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.365290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.27.46.254:33465)
0925 04:33:48.365980 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.365995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.366030 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:33:48.366653 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.366669 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.367055 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.367393 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:48.367836 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.369080 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.370094 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.370805 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.370973 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.371771 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:48.371780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:48.371797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:48.372322 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:48.372391 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:48.372417 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:48.372517 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:48.372642 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 04:33:48.372917 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":177,"threads_started":1}
I20260925 04:33:48.374650 846 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } }
I20260925 04:33:48.374905 846 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:48.374966 846 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9305df0db664db08b45093bebaa3075, State: Initialized, Role: FOLLOWER
I20260925 04:33:48.375437 846 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } }
I20260925 04:33:48.375594 846 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:48.375663 846 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:48.375757 846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:48.376935 846 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } }
I20260925 04:33:48.376988 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:33:48.377293 846 leader_election.cc:302] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9305df0db664db08b45093bebaa3075; no voters:
I20260925 04:33:48.377658 846 leader_election.cc:288] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:48.377761 851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:48.377977 851 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [term 1 LEADER]: Becoming Leader. State: Replica: b9305df0db664db08b45093bebaa3075, State: Running, Role: LEADER
I20260925 04:33:48.378314 851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } }
I20260925 04:33:48.379004 846 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:48.379464 853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9305df0db664db08b45093bebaa3075. Latest consensus state: current_term: 1 leader_uuid: "b9305df0db664db08b45093bebaa3075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } } }
I20260925 04:33:48.379607 853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.379922 852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9305df0db664db08b45093bebaa3075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9305df0db664db08b45093bebaa3075" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33465 } } }
I20260925 04:33:48.380036 852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.381268 855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:48.383886 855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:48.389626 855 catalog_manager.cc:1347] Generated new cluster ID: 18a0c3f27cf846dab43fe78ac6de4348
I20260925 04:33:48.389704 855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:48.397763 855 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:48.399044 855 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:48.410907 855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9305df0db664db08b45093bebaa3075: Generated new TSK 0
I20260925 04:33:48.411712 855 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:33:48.441452 876 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 04:33:48.441555 876 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:33:48.442142 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.419409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.27.46.254:33465)
0925 04:33:48.419548 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.419553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.419732 (+ 179us) server_negotiation.cc:407] Connection header received
0925 04:33:48.419900 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.419902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.419948 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.420026 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:33:48.420289 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.420826 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.422035 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.422249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.422323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.422639 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:33:48.441796 (+ 19157us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:33:48.442019 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 04:33:48.442224 849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 04:33:48.442332 875 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:48.419324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33465 (local address 127.0.0.1:54766)
0925 04:33:48.419638 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:48.419654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:48.419769 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:48.420054 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:48.420057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:48.420066 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:33:48.420183 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.420189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.420932 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.420935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.421918 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.421928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.422370 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.422374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.422460 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.422487 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:33:48.441991 (+ 19504us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:33:48.442161 (+ 170us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.46.254:33465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260925 04:33:48.443140 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 777
2026-09-25T04:33:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 04:33:48.483013 27835 mini_oidc.cc:152] Starting JWKS server
I20260925 04:33:48.483907 27835 webserver.cc:460] Webserver started at https://127.0.0.1:45399/ using document root <none> and password file <none>
I20260925 04:33:48.484032 27835 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:33:48.484338 27835 webserver.cc:460] Webserver started at http://127.0.0.1:40275/ using document root <none> and password file <none>
2026-09-25T04:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:48Z Disabled control of system clock
I20260925 04:33:48.492903 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43335
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:43335
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:33:48.600656 882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:48.600885 882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:48.600920 882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:48.604723 882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:48.604770 882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:48.604785 882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:48.604801 882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:48.604815 882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:33:48.604831 882 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:33:48.604844 882 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45399/jwks.json
I20260925 04:33:48.609453 882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:43335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:48.610400 882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:48.611639 882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:48.619882 891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:48.619861 888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:48.619870 889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:48.620106 882 server_base.cc:1034] running on GCE node
I20260925 04:33:48.620635 882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:48.621652 882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:48.622829 882 hybrid_clock.cc:648] HybridClock initialized: now 1790310828622813 us; error 28 us; skew 500 ppm
I20260925 04:33:48.624534 882 webserver.cc:460] Webserver started at http://127.27.46.254:39891/ using document root <none> and password file <none>
I20260925 04:33:48.625089 882 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:48.625180 882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:48.625392 882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:48.627386 882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "5965df2081d840018279a58aeaf30d67"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "634cb8c35ad324a5922291bdaa40831b"
server_key_iv: "c9bdc015672f84d66d9c02402c1679eb"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.627928 882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "5965df2081d840018279a58aeaf30d67"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "634cb8c35ad324a5922291bdaa40831b"
server_key_iv: "c9bdc015672f84d66d9c02402c1679eb"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.631605 882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:33:48.633813 897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:48.634961 882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:48.635074 882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "5965df2081d840018279a58aeaf30d67"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "634cb8c35ad324a5922291bdaa40831b"
server_key_iv: "c9bdc015672f84d66d9c02402c1679eb"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.635150 882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:48.662688 882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:48.663311 882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:48.663453 882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:48.670339 882 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:43335
I20260925 04:33:48.670365 950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:43335 every 8 connection(s)
I20260925 04:33:48.671263 882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:48.675118 951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:48.678521 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 882
I20260925 04:33:48.678632 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:48.678930 27835 external_mini_cluster.cc:1428] Setting key 496692e970f90e8fb808bb97806aa931
I20260925 04:33:48.680959 951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67: Bootstrap starting.
I20260925 04:33:48.683243 951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:48.684048 951 log.cc:824] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:48.686059 951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67: No bootstrap required, opened a new log
I20260925 04:33:48.688001 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.680203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.27.46.254:43335)
0925 04:33:48.680845 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.680865 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.680911 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:33:48.681633 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.681651 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.682026 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.682394 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:48.682837 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.683889 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.684875 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.685341 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.685531 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.686434 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:48.686439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:48.686451 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:48.686825 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:48.686877 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:48.686895 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:48.687002 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:48.687129 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:33:48.687362 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":124,"threads_started":1}
I20260925 04:33:48.689478 951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } }
I20260925 04:33:48.689823 951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:48.689911 951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5965df2081d840018279a58aeaf30d67, State: Initialized, Role: FOLLOWER
I20260925 04:33:48.690369 951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } }
I20260925 04:33:48.690531 951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:48.690616 951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:48.690725 951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:48.690896 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:33:48.691785 951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } }
I20260925 04:33:48.692137 951 leader_election.cc:302] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5965df2081d840018279a58aeaf30d67; no voters:
I20260925 04:33:48.692487 951 leader_election.cc:288] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:48.692587 957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:48.692838 957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [term 1 LEADER]: Becoming Leader. State: Replica: 5965df2081d840018279a58aeaf30d67, State: Running, Role: LEADER
I20260925 04:33:48.693131 957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } }
I20260925 04:33:48.693718 951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:48.694634 959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5965df2081d840018279a58aeaf30d67. Latest consensus state: current_term: 1 leader_uuid: "5965df2081d840018279a58aeaf30d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } } }
I20260925 04:33:48.694808 959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.695107 958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5965df2081d840018279a58aeaf30d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5965df2081d840018279a58aeaf30d67" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43335 } } }
I20260925 04:33:48.695192 966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:48.695206 958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.698199 966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:48.703730 966 catalog_manager.cc:1347] Generated new cluster ID: afb8507c24ff4e868e23815b872b9516
I20260925 04:33:48.703788 966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:48.718029 966 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:48.719205 966 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:48.727289 966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5965df2081d840018279a58aeaf30d67: Generated new TSK 0
I20260925 04:33:48.728055 966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:33:48.745965 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.741241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.27.46.254:43335)
0925 04:33:48.741353 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.741357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.741669 (+ 312us) server_negotiation.cc:407] Connection header received
0925 04:33:48.741885 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.741887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.741932 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.742039 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:33:48.742296 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.742830 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.743746 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.743973 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.744078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.744382 (+ 304us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:33:48.745573 (+ 1191us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:33:48.745826 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 04:33:48.746037 955 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 04:33:48.746352 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:48.741178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:43335 (local address 127.0.0.1:47800)
0925 04:33:48.741576 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:48.741587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:48.741741 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:48.742040 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:48.742044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:48.742057 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:33:48.742187 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.742193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.742954 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.742957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.743589 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:48.743609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.744100 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.744104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:48.744191 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.744212 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:33:48.745730 (+ 1518us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:33:48.745879 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.46.254:43335: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":95,"threads_started":1}
I20260925 04:33:48.746953 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 882
E20260925 04:33:48.757447 982 webserver.cc:533] Webserver: sslize failed:
E20260925 04:33:48.757565 982 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T04:33:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T04:33:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:48Z Disabled control of system clock
I20260925 04:33:48.795547 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:37799
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:37799
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/testchainca.cert with env {}
W20260925 04:33:48.905248 986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:48.905478 986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:48.905531 986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:48.909409 986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:48.909461 986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:48.909476 986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:48.909534 986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:48.909595 986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:33:48.909631 986 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:33:48.909653 986 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 04:33:48.914830 986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:37799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:48.915817 986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:48.916788 986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:48.925586 991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:48.925537 992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:48.925580 986 server_base.cc:1034] running on GCE node
W20260925 04:33:48.925550 994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:48.926359 986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:48.927418 986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:48.928581 986 hybrid_clock.cc:648] HybridClock initialized: now 1790310828928557 us; error 35 us; skew 500 ppm
I20260925 04:33:48.930478 986 webserver.cc:460] Webserver started at http://127.27.46.254:38207/ using document root <none> and password file <none>
I20260925 04:33:48.931124 986 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:48.931214 986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:48.931411 986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:48.933386 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "9dbe75badd6442b0bfa5f596ae453fb8"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "4a0581c27911818a92f2ccd7f2abdc08"
server_key_iv: "d0daf7c3ab8d7225671058189eeca88f"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.934095 986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "9dbe75badd6442b0bfa5f596ae453fb8"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "4a0581c27911818a92f2ccd7f2abdc08"
server_key_iv: "d0daf7c3ab8d7225671058189eeca88f"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.937718 986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:48.939985 1000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:48.941159 986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:48.941306 986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "9dbe75badd6442b0bfa5f596ae453fb8"
format_stamp: "Formatted at 2026-09-25 04:33:48 on dist-test-slave-r8r3"
server_key: "4a0581c27911818a92f2ccd7f2abdc08"
server_key_iv: "d0daf7c3ab8d7225671058189eeca88f"
server_key_version: "encryptionkey@0"
I20260925 04:33:48.941413 986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:48.954806 986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:48.955487 986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:48.955696 986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:48.962491 986 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:37799
I20260925 04:33:48.962505 1054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:37799 every 8 connection(s)
I20260925 04:33:48.963413 986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:48.966199 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:48.971048 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 986
I20260925 04:33:48.971182 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:48.971541 27835 external_mini_cluster.cc:1428] Setting key 602fabe8533baba0b8d8e6fdd881f622
I20260925 04:33:48.971648 1055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: Bootstrap starting.
I20260925 04:33:48.974155 1055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:48.974886 1055 log.cc:824] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:48.976689 1055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: No bootstrap required, opened a new log
I20260925 04:33:48.980094 1055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } }
I20260925 04:33:48.980335 1055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:48.980412 1055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbe75badd6442b0bfa5f596ae453fb8, State: Initialized, Role: FOLLOWER
I20260925 04:33:48.980769 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:48.972925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.27.46.254:37799)
0925 04:33:48.973616 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:48.973633 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:48.973670 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:33:48.974343 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:48.974361 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:48.974751 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:48.975081 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:48.975489 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.976421 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.977390 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:48.977920 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:48.978103 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:48.978945 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:48.978950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:48.978962 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:33:48.979341 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:48.979393 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:33:48.979412 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:48.979518 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:48.979678 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:33:48.979926 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260925 04:33:48.980916 1055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } }
I20260925 04:33:48.981087 1055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:48.981158 1055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:48.981245 1055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:48.982192 1055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } }
I20260925 04:33:48.982574 1055 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbe75badd6442b0bfa5f596ae453fb8; no voters:
I20260925 04:33:48.982897 1055 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:48.982959 1060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:48.983141 1060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 9dbe75badd6442b0bfa5f596ae453fb8, State: Running, Role: LEADER
I20260925 04:33:48.983418 1060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } }
I20260925 04:33:48.983624 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:33:48.984120 1055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:48.984571 1062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } } }
I20260925 04:33:48.984701 1062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:48.985387 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbe75badd6442b0bfa5f596ae453fb8. Latest consensus state: current_term: 1 leader_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbe75badd6442b0bfa5f596ae453fb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 37799 } } }
I20260925 04:33:48.985525 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8 [sys.catalog]: This master's current role is: LEADER
W20260925 04:33:48.988701 1075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:33:48.988765 1075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:33:48.988847 1065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:48.989576 1065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:48.994647 1065 catalog_manager.cc:1347] Generated new cluster ID: fe92a72fec10430aafc1edc04de85bcd
I20260925 04:33:48.994714 1065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:49.014142 1065 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:49.015322 1065 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:49.025810 1065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbe75badd6442b0bfa5f596ae453fb8: Generated new TSK 0
I20260925 04:33:49.026592 1065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:33:49.078197 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:49.042338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.27.46.254:37799)
0925 04:33:49.042474 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:49.042479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:49.042689 (+ 210us) server_negotiation.cc:407] Connection header received
0925 04:33:49.042812 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:49.042815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:49.042872 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:49.042993 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:33:49.043196 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:49.044036 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:49.044947 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:49.045297 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:49.045400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:49.045763 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:33:49.077731 (+ 31968us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:33:49.078048 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:33:49.078290 1058 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58128: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 04:33:49.078343 1084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:49.042246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:37799 (local address 127.0.0.1:58128)
0925 04:33:49.042606 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:49.042617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:49.042710 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:49.042995 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:49.042997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:49.043003 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:33:49.043099 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:49.043104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:49.044137 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:49.044139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:49.044832 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:49.044841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:49.045418 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:49.045421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:49.045487 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:49.045557 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:33:49.077978 (+ 32421us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:33:49.078153 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.46.254:37799: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260925 04:33:49.079051 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 986
2026-09-25T04:33:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (311 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:49 dist-test-slave-r8r3 krb5kdc[1097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:49 dist-test-slave-r8r3 krb5kdc[1097](info): set up 2 sockets
Sep 25 04:33:49 dist-test-slave-r8r3 krb5kdc[1097](info): commencing operation
krb5kdc: starting...
W20260925 04:33:50.393853 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:50 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:33:50.518980 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35313
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:50.628693 1110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:50.629015 1110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:50.629109 1110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:50.633126 1110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:50.633220 1110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:50.633255 1110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:50.633296 1110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:50.633327 1110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:50.638811 1110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:50.639972 1110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:50.641054 1110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:50.649729 1115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:50.649749 1110 server_base.cc:1034] running on GCE node
W20260925 04:33:50.649729 1118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:50.649933 1116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:50.650596 1110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:50.651719 1110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:50.652904 1110 hybrid_clock.cc:648] HybridClock initialized: now 1790310830652886 us; error 31 us; skew 500 ppm
Sep 25 04:33:50 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310830, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:50.661577 1110 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:33:50.662642 1110 webserver.cc:460] Webserver started at http://127.27.46.254:44469/ using document root <none> and password file <none>
I20260925 04:33:50.663268 1110 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:50.663362 1110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:50.663616 1110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:50.665858 1110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "d0c3d37042dd4d6fb0289a99cb52fa23"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "78fac1cfd4acf82f3809a66d4c490c76"
server_key_iv: "7d3a5df63ca52b7e6b148aa9ded8fb3e"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.666613 1110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "d0c3d37042dd4d6fb0289a99cb52fa23"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "78fac1cfd4acf82f3809a66d4c490c76"
server_key_iv: "7d3a5df63ca52b7e6b148aa9ded8fb3e"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.670653 1110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:33:50.673272 1125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:50.674634 1110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:50.674780 1110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "d0c3d37042dd4d6fb0289a99cb52fa23"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "78fac1cfd4acf82f3809a66d4c490c76"
server_key_iv: "7d3a5df63ca52b7e6b148aa9ded8fb3e"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.674904 1110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:50.692061 1110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:50.695575 1110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:50.695794 1110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:50.703248 1110 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35313
I20260925 04:33:50.703269 1179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35313 every 8 connection(s)
I20260925 04:33:50.704255 1110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:50.705926 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1110
I20260925 04:33:50.706051 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:50.706368 27835 external_mini_cluster.cc:1428] Setting key 52d0ebe5fe86d20512238c476663265c
I20260925 04:33:50.707566 1180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:50.714231 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23: Bootstrap starting.
Sep 25 04:33:50 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:50.716889 1180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:50.717825 1180 log.cc:824] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:50.720110 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23: No bootstrap required, opened a new log
I20260925 04:33:50.721153 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:50.707819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.27.46.254:35313)
0925 04:33:50.708258 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:50.708274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:50.708310 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:33:50.708959 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:50.708976 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:50.709297 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:50.709653 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:50.710579 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:50.711600 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:50.712805 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:50.713280 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:50.713408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:50.716256 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:50.716261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:50.716275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:50.716304 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:50.718236 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:50.718722 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:50.718727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:50.718731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:50.718838 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:50.719172 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:50.719175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:50.719176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:50.719577 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:50.719736 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:50.720080 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:33:50.720412 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":154,"threads_started":1}
I20260925 04:33:50.723436 1180 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } }
I20260925 04:33:50.723726 1180 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:50.723831 1180 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c3d37042dd4d6fb0289a99cb52fa23, State: Initialized, Role: FOLLOWER
I20260925 04:33:50.724267 1180 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } }
I20260925 04:33:50.724416 1180 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:50.724502 1180 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:50.724622 1180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:50.725584 1180 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } }
I20260925 04:33:50.725914 1180 leader_election.cc:302] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0c3d37042dd4d6fb0289a99cb52fa23; no voters:
I20260925 04:33:50.726243 1180 leader_election.cc:288] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:50.726397 1185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:50.726655 1185 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [term 1 LEADER]: Becoming Leader. State: Replica: d0c3d37042dd4d6fb0289a99cb52fa23, State: Running, Role: LEADER
I20260925 04:33:50.726999 1185 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } }
I20260925 04:33:50.727483 1180 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:50.728677 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c3d37042dd4d6fb0289a99cb52fa23. Latest consensus state: current_term: 1 leader_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } } }
I20260925 04:33:50.728824 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:50.729115 1186 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0c3d37042dd4d6fb0289a99cb52fa23" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35313 } } }
I20260925 04:33:50.729224 1186 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:50.729368 1195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:50.732095 1195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:50.737226 1195 catalog_manager.cc:1347] Generated new cluster ID: a407155b354e4d3db1136fffc3b675f7
I20260925 04:33:50.737291 1195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:50.767742 1195 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:50.769094 1195 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:50.783704 1195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0c3d37042dd4d6fb0289a99cb52fa23: Generated new TSK 0
I20260925 04:33:50.784458 1195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:33:50.817317 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:50.934188 1207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:50.934432 1207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:50.934473 1207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:50.938550 1207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:50.938603 1207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:50.938671 1207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:50.943884 1207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:50.944929 1207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:50.946069 1207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:50.955590 1215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:50.955698 1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:50.955718 1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:50.955749 1207 server_base.cc:1034] running on GCE node
I20260925 04:33:50.956282 1207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:50.956908 1207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:50.958122 1207 hybrid_clock.cc:648] HybridClock initialized: now 1790310830958106 us; error 25 us; skew 500 ppm
Sep 25 04:33:50 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310830, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:50.960839 1207 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:33:50.961730 1207 webserver.cc:460] Webserver started at http://127.27.46.193:38875/ using document root <none> and password file <none>
I20260925 04:33:50.962236 1207 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:50.962291 1207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:50.962460 1207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:50.964504 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "eb7761e73aaf43b88c8692a0f2ba6f07"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "6dbacf6732317ba7fec4157325dcd975"
server_key_iv: "5762668faef8845225c4ead70f55ef25"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.965008 1207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7761e73aaf43b88c8692a0f2ba6f07"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "6dbacf6732317ba7fec4157325dcd975"
server_key_iv: "5762668faef8845225c4ead70f55ef25"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.968647 1207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:50.971011 1222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:50.972311 1207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:50.972411 1207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "eb7761e73aaf43b88c8692a0f2ba6f07"
format_stamp: "Formatted at 2026-09-25 04:33:50 on dist-test-slave-r8r3"
server_key: "6dbacf6732317ba7fec4157325dcd975"
server_key_iv: "5762668faef8845225c4ead70f55ef25"
server_key_version: "encryptionkey@0"
I20260925 04:33:50.972497 1207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:50.985872 1207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:50.988570 1207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:50.988718 1207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:50.989257 1207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:50.990031 1207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:50.990075 1207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:50.990114 1207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:50.990136 1207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:50.998819 1207 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:43939
I20260925 04:33:50.998850 1337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:43939 every 8 connection(s)
I20260925 04:33:50.999727 1207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:51.004060 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1207
I20260925 04:33:51.004199 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:51.004474 27835 external_mini_cluster.cc:1428] Setting key 4790e54d181b518dd4ee3f590ff6f35f
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310830, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:33:51.014142 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.001462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:46791 (local address 127.27.46.254:35313)
0925 04:33:51.001647 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.001651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.002428 (+ 777us) server_negotiation.cc:407] Connection header received
0925 04:33:51.003885 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.003889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.003952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.004035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.005567 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.006268 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.007407 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.007723 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.007805 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.010501 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.010505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.010507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.010540 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.011998 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.012503 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.012508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.012510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.012589 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.012916 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.012921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.012923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.013104 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.013212 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.013835 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.013965 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 04:33:51.014952 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.001850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35313 (local address 127.27.46.193:46791)
0925 04:33:51.002278 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:51.002322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:51.003656 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:51.004282 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:51.004292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:51.004708 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:51.005375 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.005393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.006399 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.006402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.007282 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.007290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.007882 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.007888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.008109 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.008712 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:51.008737 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:51.010336 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:51.012125 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.012131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.012145 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.012396 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.012682 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.012685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.012687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.012821 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.013211 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:51.013218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:51.013550 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 04:33:51.013781 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 04:33:51.014114 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260925 04:33:51.016085 1338 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35313
I20260925 04:33:51.016402 1338 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:51.016960 1338 heartbeater.cc:511] Master 127.27.46.254:35313 requested a full tablet report, sending...
I20260925 04:33:51.018570 1142 ts_manager.cc:194] Registered new tserver with Master: eb7761e73aaf43b88c8692a0f2ba6f07 (127.27.46.193:43939)
I20260925 04:33:51.019704 1142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:46791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:33:51.033329 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:51.144733 1343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:51.144999 1343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:51.145043 1343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:51.149040 1343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:51.149106 1343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:51.149225 1343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:51.154464 1343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:51.155510 1343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:51.156539 1343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:51.166021 1351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:51.166093 1343 server_base.cc:1034] running on GCE node
W20260925 04:33:51.166049 1349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:51.166030 1348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:51.166707 1343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:51.167313 1343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:51.168497 1343 hybrid_clock.cc:648] HybridClock initialized: now 1790310831168483 us; error 24 us; skew 500 ppm
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.171294 1343 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:33:51.172178 1343 webserver.cc:460] Webserver started at http://127.27.46.194:42299/ using document root <none> and password file <none>
I20260925 04:33:51.172756 1343 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:51.172848 1343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:51.173091 1343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:51.175360 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "05152ac2c3b8468a80464728c1c99778"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "70dee814e8c8f4606dc89b8876e78677"
server_key_iv: "467972669b3ce4a9a27c0e8d10913579"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.175989 1343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "05152ac2c3b8468a80464728c1c99778"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "70dee814e8c8f4606dc89b8876e78677"
server_key_iv: "467972669b3ce4a9a27c0e8d10913579"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.179940 1343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 04:33:51.182412 1358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.183617 1343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:51.183770 1343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "05152ac2c3b8468a80464728c1c99778"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "70dee814e8c8f4606dc89b8876e78677"
server_key_iv: "467972669b3ce4a9a27c0e8d10913579"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.183881 1343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:51.202337 1343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:51.205281 1343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:51.205528 1343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:51.206184 1343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:51.207036 1343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:51.207104 1343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.207173 1343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:51.207224 1343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.216082 1343 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:45543
I20260925 04:33:51.216112 1475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:45543 every 8 connection(s)
I20260925 04:33:51.216941 1343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:51.220072 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1343
I20260925 04:33:51.220207 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:51.220629 27835 external_mini_cluster.cc:1428] Setting key 5af4c23ec2e2de4a47e2b1a25ccdac5d
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:33:51.232471 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.218692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:45429 (local address 127.27.46.254:35313)
0925 04:33:51.218875 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.218880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.219659 (+ 779us) server_negotiation.cc:407] Connection header received
0925 04:33:51.221544 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.221548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.221611 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.221721 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.223247 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.223779 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.225157 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.225394 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.225531 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.228569 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.228572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.228574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.228597 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.230273 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.230880 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.230883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.230884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.230951 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.231350 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.231352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.231353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.231516 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.231588 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.232251 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.232352 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:33:51.233347 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.219054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35313 (local address 127.27.46.194:45429)
0925 04:33:51.219498 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:51.219545 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:51.221280 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:51.221935 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:51.221943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:51.222446 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:51.223083 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.223104 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.223917 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.223924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.225022 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.225032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.225577 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.225581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.225820 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.226430 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:51.226456 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:51.228384 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:51.230406 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.230416 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.230433 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.230780 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.231046 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.231052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.231056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.231252 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.231623 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:51.231630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:51.231983 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 04:33:51.232200 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 04:33:51.232456 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:33:51.234880 1476 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35313
I20260925 04:33:51.235136 1476 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:51.235648 1476 heartbeater.cc:511] Master 127.27.46.254:35313 requested a full tablet report, sending...
I20260925 04:33:51.236789 1142 ts_manager.cc:194] Registered new tserver with Master: 05152ac2c3b8468a80464728c1c99778 (127.27.46.194:45543)
I20260925 04:33:51.237730 1142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:45429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:33:51.249387 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:51.359670 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:51.359944 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:51.359988 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:51.364120 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:51.364176 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:51.364280 1481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:51.369328 1481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:51.370427 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:51.371392 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:51.381590 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:51.381690 1481 server_base.cc:1034] running on GCE node
W20260925 04:33:51.381593 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:51.381596 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:51.382334 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:51.382932 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:51.384114 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790310831384084 us; error 51 us; skew 500 ppm
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.386945 1481 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:33:51.387833 1481 webserver.cc:460] Webserver started at http://127.27.46.195:38665/ using document root <none> and password file <none>
I20260925 04:33:51.388319 1481 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:51.388372 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:51.388537 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:51.390625 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "c92593ca863340ac8b25e4155373fca0"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "777fac70ef3d347d6f9fc8c0783eb22a"
server_key_iv: "e3f5c73a2eac05d84dbbc1bf88cf3805"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.391110 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "c92593ca863340ac8b25e4155373fca0"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "777fac70ef3d347d6f9fc8c0783eb22a"
server_key_iv: "e3f5c73a2eac05d84dbbc1bf88cf3805"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.394551 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:51.396922 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.398141 1481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:51.398258 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "c92593ca863340ac8b25e4155373fca0"
format_stamp: "Formatted at 2026-09-25 04:33:51 on dist-test-slave-r8r3"
server_key: "777fac70ef3d347d6f9fc8c0783eb22a"
server_key_iv: "e3f5c73a2eac05d84dbbc1bf88cf3805"
server_key_version: "encryptionkey@0"
I20260925 04:33:51.398357 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:51.419162 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:51.422606 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:51.422824 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:51.423458 1481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:51.424312 1481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:51.424357 1481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.424396 1481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:51.424453 1481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:51.433667 1481 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:42603
I20260925 04:33:51.433688 1612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:42603 every 8 connection(s)
I20260925 04:33:51.434576 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:51.435565 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1481
I20260925 04:33:51.435680 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:51.435994 27835 external_mini_cluster.cc:1428] Setting key 5d55865ac5171e5745b5e2ea52149800
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:33:51.448779 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.436755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:53131 (local address 127.27.46.254:35313)
0925 04:33:51.436935 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.436940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.437899 (+ 959us) server_negotiation.cc:407] Connection header received
0925 04:33:51.439471 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.439474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.439532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.439618 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.440816 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.441353 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.442547 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.442770 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.442884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.445286 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.445289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.445291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.445316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.446845 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.447357 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.447359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.447360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.447431 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.447732 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.447734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.447735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.447902 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.447975 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.448475 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.448607 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:33:51.449860 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.437175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35313 (local address 127.27.46.195:53131)
0925 04:33:51.437761 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:51.437804 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:51.439229 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:51.439824 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:51.439831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:51.440194 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:51.440676 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.440689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.441481 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.441485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.442402 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:51.442411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.442924 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.442929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:51.443098 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.443647 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:51.443662 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:51.445098 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:51.446967 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.446988 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.447001 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.447263 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.447514 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:51.447516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:51.447517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:51.447648 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:51.448000 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:51.448005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:51.448230 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 04:33:51.448428 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 04:33:51.448761 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":200,"threads_started":1}
I20260925 04:33:51.450946 1613 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35313
I20260925 04:33:51.451205 1613 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:51.451690 1613 heartbeater.cc:511] Master 127.27.46.254:35313 requested a full tablet report, sending...
I20260925 04:33:51.452679 1142 ts_manager.cc:194] Registered new tserver with Master: c92593ca863340ac8b25e4155373fca0 (127.27.46.195:42603)
I20260925 04:33:51.453254 1142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:53131
I20260925 04:33:51.461946 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/test-admin.keytab.
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.478320 27835 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 04:33:51.481199 1620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:51.479858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.27.46.193:43939)
0925 04:33:51.480138 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.480152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.480175 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:33:51.480225 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.480229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.480422 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.480569 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.481007 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43340: BlockingRecv error: recv got EOF from 127.0.0.1:43340 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
W20260925 04:33:51.481235 27835 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.482328 27835 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.483225 27835 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:51.492606 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.484102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.27.46.193:43939)
0925 04:33:51.484224 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.484231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.484348 (+ 117us) server_negotiation.cc:407] Connection header received
0925 04:33:51.484432 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.484434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.484470 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.484533 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.485005 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.485851 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.486710 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.487103 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.487181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.489402 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.489411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.489423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.489445 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.490907 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.491288 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.491293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.491294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.491365 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.491627 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.491630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.491631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.491892 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.492074 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.492232 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.492448 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 04:33:51.496491 1620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:51.495524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.27.46.193:43939)
0925 04:33:51.495651 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.495654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.495683 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:33:51.495805 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.495807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.495840 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.495906 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.496372 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43346: BlockingRecv error: recv got EOF from 127.0.0.1:43346 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 04:33:51.496579 27835 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.497707 27835 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.498615 27835 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:51.507777 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.499406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.27.46.193:43939)
0925 04:33:51.499535 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.499540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.499650 (+ 110us) server_negotiation.cc:407] Connection header received
0925 04:33:51.499801 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.499805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.499855 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.499956 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.500474 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.501124 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.502104 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.502356 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.502477 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.504732 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.504736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.504738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.504760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.506013 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.506496 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.506500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.506502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.506589 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.506914 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.506917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.506921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.507091 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.507236 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.507502 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.507622 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 04:33:51.510628 1620 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:51.509576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.27.46.193:43939)
0925 04:33:51.509701 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.509705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.509787 (+ 82us) server_negotiation.cc:407] Connection header received
0925 04:33:51.509939 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.509943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.509985 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.510091 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.510502 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43368: BlockingRecv error: recv got EOF from 127.0.0.1:43368 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 04:33:51.510784 27835 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790310801918616-27835-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.511735 27835 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:51.512574 27835 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:33:51.520728 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:51.513309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.27.46.193:43939)
0925 04:33:51.513414 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:51.513417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:51.513643 (+ 226us) server_negotiation.cc:407] Connection header received
0925 04:33:51.513716 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:51.513718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:51.513751 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:51.513815 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:51.514221 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.514643 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.515351 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:51.515618 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:51.515706 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:51.517934 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:51.517938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:51.517939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:51.517959 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:51.519284 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.519669 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.519672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.519673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.519741 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:51.520009 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:51.520016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:51.520017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:51.520158 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:51.520247 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:51.520439 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 04:33:51.520559 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:33:51.521715 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1207
I20260925 04:33:51.528754 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1343
I20260925 04:33:51.534461 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1481
I20260925 04:33:51.540328 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1110
2026-09-25T04:33:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1633](info): set up 2 sockets
Sep 25 04:33:51 dist-test-slave-r8r3 krb5kdc[1633](info): commencing operation
krb5kdc: starting...
W20260925 04:33:52.577584 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:52 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab.
Entry for principal oryx/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab.
I20260925 04:33:52.697969 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46331
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab
--principal=oryx/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:52.807842 1646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:52.808128 1646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:52.808168 1646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:52.812146 1646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:52.812202 1646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:52.812224 1646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:52.812243 1646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:52.812314 1646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:52.817201 1646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab
--principal=oryx/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:52.818257 1646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:52.819298 1646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:52.827843 1651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:52.827852 1654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:52.827898 1646 server_base.cc:1034] running on GCE node
W20260925 04:33:52.827879 1652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:52.828759 1646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:52.829864 1646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:52.831024 1646 hybrid_clock.cc:648] HybridClock initialized: now 1790310832830997 us; error 39 us; skew 500 ppm
Sep 25 04:33:52 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:52.833745 1646 init.cc:377] Logged in from keytab as oryx/127.27.46.254@KRBTEST.COM (short username oryx)
I20260925 04:33:52.834638 1646 webserver.cc:460] Webserver started at http://127.27.46.254:42729/ using document root <none> and password file <none>
I20260925 04:33:52.835182 1646 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:52.835273 1646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:52.835507 1646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:52.837565 1646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "6407d49569d8484182db8b9084d335b8"
format_stamp: "Formatted at 2026-09-25 04:33:52 on dist-test-slave-r8r3"
server_key: "e3edd50b4fc3e5caed3875a4bf358a23"
server_key_iv: "60a68b05f44d2e84ebbea7e43c39d60c"
server_key_version: "encryptionkey@0"
I20260925 04:33:52.838137 1646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "6407d49569d8484182db8b9084d335b8"
format_stamp: "Formatted at 2026-09-25 04:33:52 on dist-test-slave-r8r3"
server_key: "e3edd50b4fc3e5caed3875a4bf358a23"
server_key_iv: "60a68b05f44d2e84ebbea7e43c39d60c"
server_key_version: "encryptionkey@0"
I20260925 04:33:52.841886 1646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:33:52.844182 1661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:52.845329 1646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:52.845479 1646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "6407d49569d8484182db8b9084d335b8"
format_stamp: "Formatted at 2026-09-25 04:33:52 on dist-test-slave-r8r3"
server_key: "e3edd50b4fc3e5caed3875a4bf358a23"
server_key_iv: "60a68b05f44d2e84ebbea7e43c39d60c"
server_key_version: "encryptionkey@0"
I20260925 04:33:52.845641 1646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:52.860209 1646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:52.862948 1646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:52.863147 1646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:52.870347 1646 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:46331
I20260925 04:33:52.870647 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:46331 every 8 connection(s)
I20260925 04:33:52.871349 1646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:52.873792 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1646
I20260925 04:33:52.873937 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:52.874274 27835 external_mini_cluster.cc:1428] Setting key c9c7ff2165e9cfe0c7125f8e951fa009
I20260925 04:33:52.874307 1714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:52.880658 1714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Bootstrap starting.
Sep 25 04:33:52 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:52.883244 1714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:52.884137 1714 log.cc:824] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:52.886359 1714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: No bootstrap required, opened a new log
I20260925 04:33:52.888310 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:52.875664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.27.46.254:46331)
0925 04:33:52.876088 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:52.876102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:52.876135 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:33:52.876922 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:52.876940 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:52.877277 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:52.877638 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:52.878443 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:52.879431 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:52.880276 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:52.880807 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:52.880986 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:52.883746 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:52.883751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:52.883763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:52.883790 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:52.885562 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:52.886037 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:52.886041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:52.886045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:52.886146 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:52.886450 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:52.886456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:52.886458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:52.886844 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:52.886974 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:52.887314 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 04:33:52.887616 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260925 04:33:52.890085 1714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } }
I20260925 04:33:52.890286 1714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:52.890419 1714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6407d49569d8484182db8b9084d335b8, State: Initialized, Role: FOLLOWER
I20260925 04:33:52.890836 1714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } }
I20260925 04:33:52.890988 1714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:52.891049 1714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:52.891166 1714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:52.892164 1714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } }
I20260925 04:33:52.892509 1714 leader_election.cc:302] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6407d49569d8484182db8b9084d335b8; no voters:
I20260925 04:33:52.892844 1714 leader_election.cc:288] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:52.892927 1719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:52.893150 1719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 LEADER]: Becoming Leader. State: Replica: 6407d49569d8484182db8b9084d335b8, State: Running, Role: LEADER
I20260925 04:33:52.893457 1719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } }
I20260925 04:33:52.894112 1714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:52.895078 1721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } }
I20260925 04:33:52.895084 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6407d49569d8484182db8b9084d335b8. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } }
I20260925 04:33:52.895262 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:52.895545 1721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:52.895629 1729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:52.898729 1729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:52.904796 1729 catalog_manager.cc:1347] Generated new cluster ID: d7c2849ecbd04a16ae416b8451be5db5
I20260925 04:33:52.904871 1729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:52.914482 1729 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:52.915329 1729 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:52.929534 1729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Generated new TSK 0
I20260925 04:33:52.930251 1729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
I20260925 04:33:52.969683 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:53.098831 1741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:53.099068 1741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:53.099108 1741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:53.102916 1741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:53.102968 1741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:53.103020 1741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:53.108052 1741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:53.109001 1741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:53.109994 1741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:53.119464 1749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.119580 1746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.119580 1747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:53.119779 1741 server_base.cc:1034] running on GCE node
I20260925 04:33:53.120271 1741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:53.120968 1741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:53.122143 1741 hybrid_clock.cc:648] HybridClock initialized: now 1790310833122122 us; error 30 us; skew 500 ppm
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:53.125119 1741 init.cc:377] Logged in from keytab as oryx/127.27.46.193@KRBTEST.COM (short username oryx)
I20260925 04:33:53.126127 1741 webserver.cc:460] Webserver started at http://127.27.46.193:33531/ using document root <none> and password file <none>
I20260925 04:33:53.126643 1741 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.126703 1741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.126878 1741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:53.128878 1741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "3cf2875e42214818b62f43e6565e1acc"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "fdf456f0d5838e6d31332ff0282a8b49"
server_key_iv: "e844fdd29943dbdcc71a65eed3e12a40"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.129441 1741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "3cf2875e42214818b62f43e6565e1acc"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "fdf456f0d5838e6d31332ff0282a8b49"
server_key_iv: "e844fdd29943dbdcc71a65eed3e12a40"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.133782 1741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:33:53.136330 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.137681 1741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:53.137807 1741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "3cf2875e42214818b62f43e6565e1acc"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "fdf456f0d5838e6d31332ff0282a8b49"
server_key_iv: "e844fdd29943dbdcc71a65eed3e12a40"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.137899 1741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.149354 1741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:53.152580 1741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:53.152788 1741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:53.153383 1741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:53.154225 1741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:53.154273 1741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.154342 1741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:53.154374 1741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.163951 1741 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44017
I20260925 04:33:53.163977 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44017 every 8 connection(s)
I20260925 04:33:53.164852 1741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:53.166649 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1741
I20260925 04:33:53.166770 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:53.167106 27835 external_mini_cluster.cc:1428] Setting key d7de7cdaffa9a4471b1905da0200a163
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.194@KRBTEST.COM" created.
I20260925 04:33:53.180862 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.166831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:36711 (local address 127.27.46.254:46331)
0925 04:33:53.166980 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.166984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.168098 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 04:33:53.169706 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.169710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.169762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.169838 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.171440 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.171972 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.173811 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.174033 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.174113 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.176911 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.176914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.176916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.176939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.178542 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.179060 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.179065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.179068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.179183 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.179568 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.179574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.179577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.179828 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.179938 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.180554 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.180682 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:33:53.181744 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.167347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:36711)
0925 04:33:53.167908 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.167988 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.169447 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.170018 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.170025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.170561 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.171274 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.171297 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.172111 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.172118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.173688 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.173701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.174199 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.174205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.174449 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.175092 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.175114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.176736 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.178665 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.178671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.178686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.178952 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.179326 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.179329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.179330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.179464 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.179936 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.179943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.180288 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.180496 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.180814 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260925 04:33:53.183048 1871 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:53.183303 1871 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:53.183804 1871 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:53.185573 1678 ts_manager.cc:194] Registered new tserver with Master: 3cf2875e42214818b62f43e6565e1acc (127.27.46.193:44017)
I20260925 04:33:53.187304 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:36711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
I20260925 04:33:53.196628 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:53.312873 1876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:53.313109 1876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:53.313150 1876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:53.316913 1876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:53.316959 1876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:53.317013 1876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:53.322145 1876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:53.323036 1876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:53.323967 1876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:53.332926 1884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.332957 1882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.332916 1881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:53.333099 1876 server_base.cc:1034] running on GCE node
I20260925 04:33:53.333585 1876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:53.334169 1876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:53.335363 1876 hybrid_clock.cc:648] HybridClock initialized: now 1790310833335348 us; error 24 us; skew 500 ppm
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:53.337836 1876 init.cc:377] Logged in from keytab as oryx/127.27.46.194@KRBTEST.COM (short username oryx)
I20260925 04:33:53.338649 1876 webserver.cc:460] Webserver started at http://127.27.46.194:41527/ using document root <none> and password file <none>
I20260925 04:33:53.339198 1876 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.339287 1876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.339516 1876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:53.341686 1876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "8f035ffe30674c738868384694eceefa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "6abdaef7a2bc1d60348bb3aa05b41454"
server_key_iv: "9471665c6a1b8987b4ea8e1209b49455"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.342320 1876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "8f035ffe30674c738868384694eceefa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "6abdaef7a2bc1d60348bb3aa05b41454"
server_key_iv: "9471665c6a1b8987b4ea8e1209b49455"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.346202 1876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:33:53.348631 1892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.349946 1876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:53.350117 1876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "8f035ffe30674c738868384694eceefa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "6abdaef7a2bc1d60348bb3aa05b41454"
server_key_iv: "9471665c6a1b8987b4ea8e1209b49455"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.350225 1876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.372967 1876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:53.376091 1876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:53.376310 1876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:53.376893 1876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:53.377724 1876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:53.377794 1876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.377863 1876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:53.377905 1876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.387442 1876 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:43459
I20260925 04:33:53.387461 2008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:43459 every 8 connection(s)
I20260925 04:33:53.388329 1876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:53.393019 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 1876
I20260925 04:33:53.393137 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:53.393394 27835 external_mini_cluster.cc:1428] Setting key 409784dd8896374a1ea199802f9e3e7e
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:33:53.404160 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.390262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:56299 (local address 127.27.46.254:46331)
0925 04:33:53.390414 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.390418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.391225 (+ 807us) server_negotiation.cc:407] Connection header received
0925 04:33:53.392625 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.392629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.392690 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.392803 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.394341 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.395034 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.396971 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.397395 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.397558 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.400245 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.400250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.400257 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.400293 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.402024 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.402490 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.402495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.402497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.402592 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.402922 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.402927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.402929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.403184 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.403300 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.403790 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.403934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:53.404973 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.390623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:56299)
0925 04:33:53.391074 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.391118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.392413 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.392999 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.393006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.393477 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.394129 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.394172 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.395198 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.395201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.396760 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.396793 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.397615 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.397618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.397755 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.398348 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.398364 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.400058 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.402133 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.402137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.402149 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.402383 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.402686 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.402688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.402689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.402812 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.403287 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.403292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.403513 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.403703 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.403950 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
I20260925 04:33:53.406365 2009 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:53.406599 2009 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.195@KRBTEST.COM" created.
I20260925 04:33:53.407193 2009 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:53.408357 1678 ts_manager.cc:194] Registered new tserver with Master: 8f035ffe30674c738868384694eceefa (127.27.46.194:43459)
I20260925 04:33:53.409230 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:56299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
I20260925 04:33:53.424345 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:53.533636 2015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:53.533871 2015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:53.533924 2015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:53.537914 2015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:53.537966 2015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:53.538045 2015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:53.543370 2015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:53.544387 2015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:53.545411 2015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:53.555471 2023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.555547 2020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.555526 2021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:53.555720 2015 server_base.cc:1034] running on GCE node
I20260925 04:33:53.556190 2015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:53.556838 2015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:53.558012 2015 hybrid_clock.cc:648] HybridClock initialized: now 1790310833557984 us; error 39 us; skew 500 ppm
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:53.560676 2015 init.cc:377] Logged in from keytab as oryx/127.27.46.195@KRBTEST.COM (short username oryx)
I20260925 04:33:53.561671 2015 webserver.cc:460] Webserver started at http://127.27.46.195:43809/ using document root <none> and password file <none>
I20260925 04:33:53.562219 2015 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.562330 2015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.562557 2015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:53.564606 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "eddd6246aa17499bb0c827a34aed67be"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "ca43972d529af05db9d15973f9607131"
server_key_iv: "68efd80506eac6b393ac15ac128e99f0"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.565181 2015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "eddd6246aa17499bb0c827a34aed67be"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "ca43972d529af05db9d15973f9607131"
server_key_iv: "68efd80506eac6b393ac15ac128e99f0"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.568893 2015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:53.571312 2031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.572520 2015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:33:53.572655 2015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "eddd6246aa17499bb0c827a34aed67be"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "ca43972d529af05db9d15973f9607131"
server_key_iv: "68efd80506eac6b393ac15ac128e99f0"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.572772 2015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.590258 2015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:53.593178 2015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:53.593379 2015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:53.594100 2015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:53.595077 2015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:53.595162 2015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.595227 2015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:53.595288 2015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.604185 2015 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:33627
I20260925 04:33:53.604212 2146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:33627 every 8 connection(s)
I20260925 04:33:53.605113 2015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:53.610708 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2015
I20260925 04:33:53.610848 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:53.611152 27835 external_mini_cluster.cc:1428] Setting key e069bd0778b0da7793fb7359d34a5b1b
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:53.618826 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.606953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:45413 (local address 127.27.46.254:46331)
0925 04:33:53.607109 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.607114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.607986 (+ 872us) server_negotiation.cc:407] Connection header received
0925 04:33:53.609279 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.609282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.609333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.609409 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.610603 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.611275 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.612902 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.613207 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.613327 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.615749 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.615754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.615756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.615791 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.617055 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.617487 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.617490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.617537 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.617642 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.617938 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.617940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.617941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.618102 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.618176 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.618621 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.618712 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:33:53.619607 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.607406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:45413)
0925 04:33:53.607841 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.607881 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.609096 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.609624 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.609630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.609969 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.610417 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.610455 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.611408 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.611415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.612774 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.612784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.613383 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.613386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.613574 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.614109 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.614125 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.615535 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.617156 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.617160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.617171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.617392 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.617731 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.617733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.617734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.617857 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.618186 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.618191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.618418 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.618571 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.618793 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260925 04:33:53.620653 2147 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:53.620982 2147 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:53.621454 2147 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:53.622459 1678 ts_manager.cc:194] Registered new tserver with Master: eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:53.623039 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:45413
I20260925 04:33:53.624845 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.253.keytab.
Entry for principal oryx/127.27.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.253.keytab.
I20260925 04:33:53.650477 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:37271
--webserver_interface=127.27.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.253.keytab
--principal=oryx/127.27.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46331,127.27.46.253:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:53.760324 2152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:53.760596 2152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:53.760635 2152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:53.764678 2152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:53.764745 2152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:53.764770 2152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:53.764788 2152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:53.764807 2152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:53.770047 2152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46331,127.27.46.253:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.253.keytab
--principal=oryx/127.27.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
W20260925 04:33:53.770236 2152 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.46.254:46331,127.27.46.253:37271'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 04:33:53.771063 2152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:53.772013 2152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:53.780668 2158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.780848 2157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.780736 2160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:53.780736 2152 server_base.cc:1034] running on GCE node
I20260925 04:33:53.781630 2152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:53.782691 2152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:53.783878 2152 hybrid_clock.cc:648] HybridClock initialized: now 1790310833783863 us; error 27 us; skew 500 ppm
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:53.786747 2152 init.cc:377] Logged in from keytab as oryx/127.27.46.253@KRBTEST.COM (short username oryx)
I20260925 04:33:53.787539 2152 webserver.cc:460] Webserver started at http://127.27.46.253:40459/ using document root <none> and password file <none>
I20260925 04:33:53.788044 2152 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.788102 2152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.788265 2152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:53.790318 2152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/instance:
uuid: "3a455d41894b401b8ccc392de5e2b3fa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "b6d89c0a5ac6f3462ae89719189f1f02"
server_key_iv: "64f13e03df694b8225efc5aa2886fa36"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.790822 2152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal/instance:
uuid: "3a455d41894b401b8ccc392de5e2b3fa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "b6d89c0a5ac6f3462ae89719189f1f02"
server_key_iv: "64f13e03df694b8225efc5aa2886fa36"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.794575 2152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:33:53.797137 2167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.798378 2152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:53.798475 2152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "3a455d41894b401b8ccc392de5e2b3fa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "b6d89c0a5ac6f3462ae89719189f1f02"
server_key_iv: "64f13e03df694b8225efc5aa2886fa36"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.798601 2152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.824358 2152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:53.827625 2152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:53.827821 2152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:53.834509 2152 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.253:37271
I20260925 04:33:53.834798 2220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.253:37271 every 8 connection(s)
I20260925 04:33:53.835455 2152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
I20260925 04:33:53.836700 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2152
I20260925 04:33:53.838560 2221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:53.843097 2221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.253@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:53.852185 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.838221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.27.46.253:37271)
0925 04:33:53.838632 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.838651 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.838837 (+ 186us) server_negotiation.cc:407] Connection header received
0925 04:33:53.839708 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.839731 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.840127 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.840486 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.841173 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.842587 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.843391 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.844085 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.844237 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.847106 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.847111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.847119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.847142 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.848923 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.849469 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.849473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.849477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.849646 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.849964 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.849968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.849972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.850467 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.850671 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.850902 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.851160 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260925 04:33:53.854401 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.845095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.27.46.254:46331)
0925 04:33:53.845244 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.845249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.845532 (+ 283us) server_negotiation.cc:407] Connection header received
0925 04:33:53.845713 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.845716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.845760 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.845841 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.846529 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.847240 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.848112 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.848365 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.848470 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.851067 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.851071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.851073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.851096 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.852576 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.853008 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.853012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.853014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.853093 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.853431 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.853434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.853435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.853741 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.853835 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.854159 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.854259 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:33:53.854413 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.845021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47170)
0925 04:33:53.845396 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.845431 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.845580 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.845889 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.845894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.846220 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.846375 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.846381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.847368 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.847371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.847982 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.847990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.848473 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.848477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.848586 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.849088 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.849103 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.850936 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.852683 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.852690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.852694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.852922 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.853216 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.853219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.853221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.853313 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.853864 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.853869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.853965 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.854102 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.854260 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260925 04:33:53.855450 2221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.253@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
I20260925 04:33:53.862661 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.855860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52538)
0925 04:33:53.855959 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.855971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.856070 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.856241 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.856243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.856355 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.856435 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.856439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.857440 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.857443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.857972 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.857980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.858366 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.858369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.858433 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.858755 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.858765 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.859841 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.861278 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.861281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.861282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.861449 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.861917 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.861919 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.861920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.861992 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.862308 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.862310 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.862376 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.862453 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.862548 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 04:33:53.862773 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.856011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.27.46.253:37271)
0925 04:33:53.856094 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.856097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.856108 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:33:53.856136 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.856137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.856176 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.856234 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.856521 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.857333 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.858072 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.858279 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.858351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.859958 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.859961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.859962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.859982 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.861174 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.861708 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.861710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.861711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.861815 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.862083 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.862085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.862086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.862231 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.862302 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.862518 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.862596 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 04:33:53.865356 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Bootstrap starting.
I20260925 04:33:53.867688 2221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:53.868358 2221 log.cc:824] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:53.870247 2221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: No bootstrap required, opened a new log
I20260925 04:33:53.873173 2221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } }
I20260925 04:33:53.873394 2221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:53.873498 2221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a455d41894b401b8ccc392de5e2b3fa, State: Initialized, Role: FOLLOWER
I20260925 04:33:53.874013 2221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } }
I20260925 04:33:53.874500 2227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } } }
I20260925 04:33:53.874642 2227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:53.875231 2221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:33:53.880143 2238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:33:53.880278 2238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:33:53.881748 2152 master_runner.cc:428] Detected that this master 3a455d41894b401b8ccc392de5e2b3fa is joining an existing cluster
I20260925 04:33:53.881806 2152 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.46.253:37271
I20260925 04:33:53.891942 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.883893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52542 (local address 127.27.46.253:37271)
0925 04:33:53.883999 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.884004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.884506 (+ 502us) server_negotiation.cc:407] Connection header received
0925 04:33:53.884689 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.884694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.884741 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.884814 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.885349 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.886122 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.887042 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.887309 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.887414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.888729 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.888736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.888738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.888764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.890325 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.890793 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.890797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.890798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.890894 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.891161 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.891164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.891166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.891317 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.891422 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.891674 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.891800 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":39}
I20260925 04:33:53.891954 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.884111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52542)
0925 04:33:53.884416 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.884430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.884564 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.884914 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.884916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.885115 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.885228 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.885233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.886256 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.886264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.886933 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.886940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.887512 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.887515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.887603 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.888167 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.888177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.888569 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.890432 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.890438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.890441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.890696 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.890991 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.890994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.890995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.891071 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.891411 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.891414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.891510 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.891614 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.891751 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":103,"spinlock_wait_cycles":7424,"thread_start_us":101,"threads_started":1}
I20260925 04:33:53.893245 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.883884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.27.46.254:46331)
0925 04:33:53.884023 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.884028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.884468 (+ 440us) server_negotiation.cc:407] Connection header received
0925 04:33:53.884613 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.884616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.884671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.884751 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.885336 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.886122 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.887403 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.887710 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.887830 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.889689 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.889693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.889696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.889723 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.891340 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.891824 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.891828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.891830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.891911 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.892327 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.892330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.892332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.892544 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.892617 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.892987 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.893107 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:33:53.893407 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.883737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47184)
0925 04:33:53.884381 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.884392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.884508 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.884787 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.884790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.885046 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.885209 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.885218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.886236 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.886240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.887271 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.887282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.888172 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.888176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.888270 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.889066 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.889083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.889534 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.891463 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.891466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.891468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.891712 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.892023 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.892026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.892028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.892226 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.892689 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.892692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.892785 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.892913 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.893036 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":55,"thread_start_us":93,"threads_started":1}
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.254@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
I20260925 04:33:53.906919 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.897720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.27.46.253:37271)
0925 04:33:53.897853 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.897858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.898227 (+ 369us) server_negotiation.cc:407] Connection header received
0925 04:33:53.898396 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.898398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.898440 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.898504 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.899244 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.899753 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.900966 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.901188 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.901269 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.903722 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.903725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.903727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.903747 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.905073 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.905664 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.905670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.905672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.905750 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.906029 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.906035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.906037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.906221 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.906331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.906655 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.906763 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:53.906948 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.897682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52546)
0925 04:33:53.898088 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.898145 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.898274 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.898563 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.898570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.898996 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.899136 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.899144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.899868 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.899872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.900854 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.900865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.901298 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.901301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.901407 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.901947 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.901962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.903570 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.905194 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.905203 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.905209 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.905543 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.905840 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.905844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.905846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.905935 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.906331 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.906339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.906463 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.906605 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.906789 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":119,"threads_started":1}
I20260925 04:33:53.908452 1678 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.46.253:37271: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa"
member_type: NON_VOTER
last_known_addr {
host: "127.27.46.253"
port: 37271
}
attrs {
promote: true
}
}
dest_uuid: "6407d49569d8484182db8b9084d335b8"
cas_config_opid_index: -1
I20260925 04:33:53.909168 1678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } }
I20260925 04:33:53.911554 2195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:53.912688 2195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 04:33:53.911033 2248 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:53.913622 2249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:53.914738 2248 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.46.253:37271
I20260925 04:33:53.914961 2248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:53.915307 2152 master.cc:537] Master@127.27.46.253:37271 shutting down...
WARNING: no policy specified for oryx/127.27.46.252@KRBTEST.COM; defaulting to no policy
I20260925 04:33:53.919198 2195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.252@KRBTEST.COM" created.
I20260925 04:33:53.920708 2251 mvcc.cc:205] Tried to move back new op lower bound from 7333113171673939968 to 7333113171532505088. Current Snapshot: MvccSnapshot[applied={T|T < 7333113171567824896}]
I20260925 04:33:53.921348 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:53.921445 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:53.921553 2258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:53.921653 2258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:53.921835 2227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6407d49569d8484182db8b9084d335b8. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:53.921895 2227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:53.922506 2248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:53.925127 2152 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Raft consensus shutting down.
I20260925 04:33:53.925391 2152 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Raft consensus is shut down!
I20260925 04:33:53.925460 2152 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: stopping tablet replica
I20260925 04:33:53.932180 2152 master.cc:552] Master@127.27.46.253:37271 shutdown complete.
I20260925 04:33:53.932263 2152 master_runner.cc:305] Clearing existing system tablet
I20260925 04:33:53.932533 2152 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.932588 2152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.933874 2152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:53.934731 2262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.934952 2152 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.935087 2152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:53.935145 2152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "3a455d41894b401b8ccc392de5e2b3fa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "b6d89c0a5ac6f3462ae89719189f1f02"
server_key_iv: "64f13e03df694b8225efc5aa2886fa36"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.937001 2152 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:53.939479 2152 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:33:53.939558 2152 log.cc:1196] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.252.keytab.
Entry for principal oryx/127.27.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.252.keytab.
I20260925 04:33:53.939885 2152 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Deleting consensus metadata
I20260925 04:33:53.940060 2152 master_runner.cc:315] Copying system tablet from 127.27.46.254:46331
I20260925 04:33:53.940821 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:35541
--webserver_interface=127.27.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.252.keytab
--principal=oryx/127.27.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:33:53.941737 2152 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: tablet copy: Beginning tablet copy session from remote peer at address 127.27.46.254:46331
I20260925 04:33:53.952275 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.942738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.27.46.254:46331)
0925 04:33:53.942877 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:53.942882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:53.943094 (+ 212us) server_negotiation.cc:407] Connection header received
0925 04:33:53.943275 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:53.943279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:53.943326 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:53.943376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:53.944032 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.944863 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.946308 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.946653 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.946755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.948247 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:53.948250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:53.948254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:53.948283 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:53.950159 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.950706 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.950709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.950711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.950799 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:53.951138 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:53.951141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:53.951143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:53.951368 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:53.951467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:53.951871 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 04:33:53.952023 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:33:53.952984 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:53.942351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47194)
0925 04:33:53.943009 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:53.943023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:53.943169 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:53.943526 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:53.943529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:53.943769 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:53.943890 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.943897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.944986 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.944992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.946168 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:53.946182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:53.946789 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:53.946795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:53.946932 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:53.947519 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:53.947539 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:53.948052 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:53.950282 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.950289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.950294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.950585 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.950909 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:53.950915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:53.950919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:53.951029 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:53.951480 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:53.951486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:53.951619 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 04:33:53.952533 (+ 914us) client_negotiation.cc:240] Negotiation successful
0925 04:33:53.952719 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":78,"threads_started":1}
I20260925 04:33:53.953660 1698 tablet_copy_service.cc:140] P 6407d49569d8484182db8b9084d335b8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3a455d41894b401b8ccc392de5e2b3fa ({username='oryx', principal='oryx/127.27.46.253@KRBTEST.COM'} at 127.0.0.1:47194)
I20260925 04:33:53.953847 1698 tablet_copy_service.cc:161] P 6407d49569d8484182db8b9084d335b8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3a455d41894b401b8ccc392de5e2b3fa at {username='oryx', principal='oryx/127.27.46.253@KRBTEST.COM'} at 127.0.0.1:47194: session id = 3a455d41894b401b8ccc392de5e2b3fa-00000000000000000000000000000000
I20260925 04:33:53.955798 1698 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:33:53.957669 2152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:33:53.960119 2152 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: tablet copy: Starting download of 0 data blocks...
I20260925 04:33:53.960302 2152 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: tablet copy: Starting download of 1 WAL segments...
I20260925 04:33:53.962266 2152 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:33:53.964252 1698 tablet_copy_service.cc:342] P 6407d49569d8484182db8b9084d335b8: Request end of tablet copy session 3a455d41894b401b8ccc392de5e2b3fa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.46.253@KRBTEST.COM'} at 127.0.0.1:47194
I20260925 04:33:53.964437 1698 tablet_copy_service.cc:433] P 6407d49569d8484182db8b9084d335b8: ending tablet copy session 3a455d41894b401b8ccc392de5e2b3fa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3a455d41894b401b8ccc392de5e2b3fa
I20260925 04:33:53.972054 2152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:53.973834 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.973963 2271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:53.974144 2273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:53.975020 2152 server_base.cc:1034] running on GCE node
I20260925 04:33:53.975212 2152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:53.975565 2152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:53.976769 2152 hybrid_clock.cc:648] HybridClock initialized: now 1790310833976755 us; error 31 us; skew 500 ppm
Sep 25 04:33:53 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310833, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:53.978543 2152 init.cc:377] Logged in from keytab as oryx/127.27.46.253@KRBTEST.COM (short username oryx)
I20260925 04:33:53.979043 2152 webserver.cc:460] Webserver started at http://127.27.46.253:44515/ using document root <none> and password file <none>
I20260925 04:33:53.979279 2152 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:53.979372 2152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:53.980736 2152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:53.982015 2281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:53.982290 2152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:53.983513 2152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "3a455d41894b401b8ccc392de5e2b3fa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "b6d89c0a5ac6f3462ae89719189f1f02"
server_key_iv: "64f13e03df694b8225efc5aa2886fa36"
server_key_version: "encryptionkey@0"
I20260925 04:33:53.983624 2152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:53.992556 2152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:53.994745 2152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:53.999491 2152 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.253:37271
I20260925 04:33:53.999748 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.253:37271 every 8 connection(s)
I20260925 04:33:54.000207 2152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
I20260925 04:33:54.002076 2334 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:33:54.003093 2334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Bootstrap starting.
I20260925 04:33:54.010569 2334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:33:54.010896 2334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: Bootstrap complete.
I20260925 04:33:54.011466 2334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } }
I20260925 04:33:54.011601 2334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3a455d41894b401b8ccc392de5e2b3fa, State: Initialized, Role: LEARNER
I20260925 04:33:54.011704 2334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } }
I20260925 04:33:54.012048 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } }
I20260925 04:33:54.012166 2334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:54.012233 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:54.015308 2348 catalog_manager.cc:1259] Loaded cluster ID: d7c2849ecbd04a16ae416b8451be5db5
I20260925 04:33:54.015472 2348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: loading cluster ID for follower catalog manager: success
I20260925 04:33:54.018095 2348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: acquiring CA information for follower catalog manager: success
I20260925 04:33:54.020761 2348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 04:33:54.076625 2268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:54.076967 2268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:54.077046 2268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:54.081000 2268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:54.081085 2268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:54.081121 2268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:54.081157 2268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:54.081192 2268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:54.086923 2268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.252.keytab
--principal=oryx/127.27.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:35541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:54.088001 2268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:54.088994 2268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:54.097311 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.097321 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.097667 2268 server_base.cc:1034] running on GCE node
W20260925 04:33:54.097819 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.098388 2268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:54.099467 2268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:54.100636 2268 hybrid_clock.cc:648] HybridClock initialized: now 1790310834100607 us; error 42 us; skew 500 ppm
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:54.103277 2268 init.cc:377] Logged in from keytab as oryx/127.27.46.252@KRBTEST.COM (short username oryx)
I20260925 04:33:54.104106 2268 webserver.cc:460] Webserver started at http://127.27.46.252:46835/ using document root <none> and password file <none>
I20260925 04:33:54.104601 2268 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.104658 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.104817 2268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:54.106807 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/instance:
uuid: "e75683aea8a34316b72bc2b39c65d087"
format_stamp: "Formatted at 2026-09-25 04:33:54 on dist-test-slave-r8r3"
server_key: "804e9a83919fccc8449410430c3185a4"
server_key_iv: "22a05b1144424d7bebcb9b96f726cd2e"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.107376 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal/instance:
uuid: "e75683aea8a34316b72bc2b39c65d087"
format_stamp: "Formatted at 2026-09-25 04:33:54 on dist-test-slave-r8r3"
server_key: "804e9a83919fccc8449410430c3185a4"
server_key_iv: "22a05b1144424d7bebcb9b96f726cd2e"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.110837 2268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:54.113144 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.114537 2268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:33:54.114634 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "e75683aea8a34316b72bc2b39c65d087"
format_stamp: "Formatted at 2026-09-25 04:33:54 on dist-test-slave-r8r3"
server_key: "804e9a83919fccc8449410430c3185a4"
server_key_iv: "22a05b1144424d7bebcb9b96f726cd2e"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.114714 2268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:54.129820 2268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:54.132411 2268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:54.132541 2268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:54.139887 2268 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.252:35541
I20260925 04:33:54.140195 2417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.252:35541 every 8 connection(s)
I20260925 04:33:54.140901 2268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
I20260925 04:33:54.143932 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:54.147714 2418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:33:54.147995 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2268
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.252@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
I20260925 04:33:54.162740 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.149888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.27.46.254:46331)
0925 04:33:54.150057 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.150064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.150456 (+ 392us) server_negotiation.cc:407] Connection header received
0925 04:33:54.151227 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.151230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.151277 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.151366 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.152462 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.153323 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.154827 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.155185 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.155319 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.158678 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.158683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.158686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.158720 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.160349 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.160971 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.160975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.160976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.161084 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.161442 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.161445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.161446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.161726 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.161827 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.162416 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.162549 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:33:54.163717 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.149745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.27.46.252:35541)
0925 04:33:54.150269 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.150284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.150315 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:33:54.151005 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.151023 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.151248 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.151411 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.152378 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.153760 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.155022 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.155821 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.155969 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.159088 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.159095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.159107 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.159138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.160840 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.161348 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.161354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.161355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.161436 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.161750 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.161752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.161753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.162141 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.162347 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.162662 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.162779 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":402,"spinlock_wait_cycles":64768,"thread_start_us":202,"threads_started":1}
I20260925 04:33:54.163717 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.149745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47208)
0925 04:33:54.150296 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.150347 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.151024 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.151446 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.151451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.151932 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.152303 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.152325 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.153465 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.153467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.154687 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.154699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.155506 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.155512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.155916 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.156808 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.156826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.158482 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.160465 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.160475 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.160486 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.160864 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.161178 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.161180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.161181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.161341 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.161828 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.161834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.162141 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.162322 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.162667 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":145,"spinlock_wait_cycles":4096,"thread_start_us":147,"threads_started":1}
I20260925 04:33:54.166662 2418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.252@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
I20260925 04:33:54.174821 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.167125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52554)
0925 04:33:54.167244 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.167258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.167332 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.167760 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.167763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.167941 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.168039 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.168045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.168956 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.168962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.169575 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.169585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.170056 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.170060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.170157 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.170592 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.170603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.171823 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.173614 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.173617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.173618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.173798 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.174058 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.174060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.174061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.174130 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.174447 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.174449 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.174530 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.174615 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.174710 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:33:54.174988 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.167264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.27.46.253:37271)
0925 04:33:54.167554 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.167557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.167569 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:33:54.167605 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.167607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.167646 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.167744 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.168157 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.168839 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.169699 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.169936 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.170051 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.171952 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.171955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.171957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.171980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.173459 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.173897 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.173899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.173900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.173976 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.174205 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.174207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.174208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.174371 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.174443 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.174711 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.174814 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260925 04:33:54.175676 2418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.252@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
I20260925 04:33:54.183135 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.176166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.0.0.1:33696)
0925 04:33:54.176290 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.176304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.176379 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.176616 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.176619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.176770 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.176867 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.176875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.177660 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.177662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.178136 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.178144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.178578 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.178580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.178635 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.178997 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.179007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.180125 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.181703 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.181709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.181711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.181901 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.182174 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.182177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.182179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.182253 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.182609 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.182612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.182694 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.182796 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.182959 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 04:33:54.183149 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.176218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.27.46.252:35541)
0925 04:33:54.176314 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.176320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.176339 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:33:54.176480 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.176483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.176519 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.176607 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.176992 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.177553 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.178230 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.178481 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.178557 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.180258 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.180262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.180264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.180283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.181597 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.181998 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.182004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.182006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.182079 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.182345 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.182348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.182349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.182515 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.182620 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.182848 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.183028 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":43}
I20260925 04:33:54.186643 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Bootstrap starting.
I20260925 04:33:54.189150 2418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:54.189850 1871 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
I20260925 04:33:54.190032 2418 log.cc:824] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:54.192124 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: No bootstrap required, opened a new log
I20260925 04:33:54.195688 2418 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } }
I20260925 04:33:54.195884 2418 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:54.195972 2418 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75683aea8a34316b72bc2b39c65d087, State: Initialized, Role: FOLLOWER
I20260925 04:33:54.196463 2418 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } }
I20260925 04:33:54.196972 2424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } } }
I20260925 04:33:54.197120 2424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:54.197799 2418 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:33:54.202979 2436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:33:54.203073 2436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:33:54.206347 2268 master_runner.cc:428] Detected that this master e75683aea8a34316b72bc2b39c65d087 is joining an existing cluster
I20260925 04:33:54.206410 2268 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.46.252:35541
I20260925 04:33:54.217295 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1741
I20260925 04:33:54.222362 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.210413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.27.46.252:35541)
0925 04:33:54.210565 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.210571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.211137 (+ 566us) server_negotiation.cc:407] Connection header received
0925 04:33:54.211310 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.211313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.211357 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.211407 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.212387 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213166 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.214772 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215075 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215139 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.216497 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.216500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.216503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.216525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.218710 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.221148 (+ 2438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.221153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.221155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.221238 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.221535 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.221540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.221541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.221718 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.221823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.222082 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.222238 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":47}
I20260925 04:33:54.222376 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.210044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.0.0.1:33712)
0925 04:33:54.211061 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.211075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.211209 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.211551 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.211554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.211993 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.212270 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.212275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213290 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.213292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.214653 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.214663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215282 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.215361 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.215910 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.215920 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.216371 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.220845 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.220847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.220849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.221043 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.221330 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.221335 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.221337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.221416 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.221813 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.221816 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.221921 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.222033 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.222181 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":136,"spinlock_wait_cycles":35968,"thread_start_us":834,"threads_started":1}
I20260925 04:33:54.227632 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:44017
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:46331
--webserver_port=33531
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:33:54.239070 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.210261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.27.46.254:46331)
0925 04:33:54.210421 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.210425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.211117 (+ 692us) server_negotiation.cc:407] Connection header received
0925 04:33:54.211344 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.211348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.211427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.211487 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.212486 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213326 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.214744 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215080 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215157 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.235107 (+ 19950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.235112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.235115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.235147 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.237240 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237763 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.237768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.237771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.237851 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.238128 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.238132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.238133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.238354 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.238442 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.238779 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.238901 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:33:54.239256 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.209968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52566)
0925 04:33:54.211050 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.211067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.211209 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.211562 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.211564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.211973 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.212263 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.212273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213489 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.213492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.214877 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.214884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215568 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.215668 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.234149 (+ 18481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.234171 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.234915 (+ 744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.237177 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.237185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.237454 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.237862 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.237870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.237968 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.238534 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.238539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.238666 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.238773 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.238950 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":70,"spinlock_wait_cycles":8576,"thread_start_us":555,"threads_started":1}
I20260925 04:33:54.239825 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.209880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.27.46.253:37271)
0925 04:33:54.210012 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.210028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.211142 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 04:33:54.211318 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.211321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.211379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.211432 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.212445 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213242 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.214987 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215341 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215410 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.235140 (+ 19730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.235146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.235152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.235193 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.237025 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237643 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.237649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.237652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.237743 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.238101 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.238107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.238110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.238378 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.238481 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.239511 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.239673 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:54.240209 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.209240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47210)
0925 04:33:54.211029 (+ 1789us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.211045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.211217 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.211705 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.211708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.212097 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.212306 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.212315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.213533 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.213536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.214603 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.214614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.215316 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.215319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.215426 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.234246 (+ 18820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.234262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.234901 (+ 639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.237380 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.237385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.237651 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.237954 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.237957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.237959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.238036 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.238499 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.238502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.238598 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.239886 (+ 1288us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.240031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"mutex_wait_us":42,"spinlock_wait_cycles":40704,"thread_start_us":197,"threads_started":1}
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310832, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.254@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
I20260925 04:33:54.253708 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.244423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.27.46.252:35541)
0925 04:33:54.244535 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.244539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.244615 (+ 76us) server_negotiation.cc:407] Connection header received
0925 04:33:54.244775 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.244777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.244839 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.244939 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.245420 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.246233 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.247355 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.247704 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.247805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.250200 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.250204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.250207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.250233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.251849 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.252346 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.252349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.252351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.252437 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.252742 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.252748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.252752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.252997 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.253105 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.253396 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.253559 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:33:54.253716 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.244333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.0.0.1:33720)
0925 04:33:54.244556 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.244571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.244658 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.244971 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.244974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.245172 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.245295 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.245302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.246360 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.246364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.247226 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.247239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.247822 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.247826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.247928 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.248426 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.248441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.250038 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.251973 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.251979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.251983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.252239 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.252537 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.252540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.252543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.252639 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.253110 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.253114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.253217 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.253328 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.253536 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 04:33:54.255131 1678 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.46.252:35541: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e75683aea8a34316b72bc2b39c65d087"
member_type: NON_VOTER
last_known_addr {
host: "127.27.46.252"
port: 35541
}
attrs {
promote: true
}
}
dest_uuid: "6407d49569d8484182db8b9084d335b8"
cas_config_opid_index: 5
I20260925 04:33:54.255748 1678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } }
I20260925 04:33:54.256845 2250 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e75683aea8a34316b72bc2b39c65d087 (127.27.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.258368 2250 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.46.252:35541
I20260925 04:33:54.258566 2250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.258672 2250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.259490 2392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:54.260854 2392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 04:33:54.261557 2268 master.cc:537] Master@127.27.46.252:35541 shutting down...
I20260925 04:33:54.266885 2246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.256904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:52582)
0925 04:33:54.262289 (+ 5385us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.262305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.262405 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.262817 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.262820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.262836 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:54.262922 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.262928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.264008 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.264011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.265270 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.265281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.266452 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.266456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.266555 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.266602 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.266674 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.266749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5303}
I20260925 04:33:54.267287 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.257155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.27.46.253:37271)
0925 04:33:54.257458 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.257462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.262492 (+ 5030us) server_negotiation.cc:407] Connection header received
0925 04:33:54.262552 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.262555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.262619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.262699 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:54.263040 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.263870 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.265430 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.266115 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.266223 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.266375 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.267089 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.267171 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260925 04:33:54.274245 2308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 04:33:54.274787 2248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:33:54.276782 1663 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 04:33:54.277287 2248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 04:33:54.279170 2308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e75683aea8a34316b72bc2b39c65d087 (127.27.46.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.280836 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6407d49569d8484182db8b9084d335b8. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.280946 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:54.281030 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: NON_VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: true } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.281097 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: LEARNER
W20260925 04:33:54.281879 1663 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 -> Peer e75683aea8a34316b72bc2b39c65d087 (127.27.46.252:35541): Couldn't send request to peer e75683aea8a34316b72bc2b39c65d087. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:33:54.282819 2268 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:54.283198 2268 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:54.283320 2248 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: attempting to promote NON_VOTER 3a455d41894b401b8ccc392de5e2b3fa to VOTER
I20260925 04:33:54.283337 2268 tablet_replica.cc:330] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: stopping tablet replica
I20260925 04:33:54.283805 2248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } }
I20260925 04:33:54.286312 2308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 LEARNER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 04:33:54.286787 2250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 04:33:54.288786 2250 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
W20260925 04:33:54.289166 1663 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 -> Peer e75683aea8a34316b72bc2b39c65d087 (127.27.46.252:35541): Couldn't send request to peer e75683aea8a34316b72bc2b39c65d087. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:33:54.290791 2250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.290900 2250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.291021 2250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.291090 2250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.291646 2308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.293131 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.293259 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:54.306774 2268 master.cc:552] Master@127.27.46.252:35541 shutdown complete.
I20260925 04:33:54.306874 2268 master_runner.cc:305] Clearing existing system tablet
I20260925 04:33:54.307139 2268 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.307214 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.308640 2268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 04:33:54.309631 2451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.309913 2268 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:54.310034 2268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:54.310068 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "e75683aea8a34316b72bc2b39c65d087"
format_stamp: "Formatted at 2026-09-25 04:33:54 on dist-test-slave-r8r3"
server_key: "804e9a83919fccc8449410430c3185a4"
server_key_iv: "22a05b1144424d7bebcb9b96f726cd2e"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.312100 2268 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:54.314564 2268 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:33:54.314653 2268 log.cc:1196] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 04:33:54.314898 2268 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Deleting consensus metadata
I20260925 04:33:54.315057 2268 master_runner.cc:315] Copying system tablet from 127.27.46.254:46331
I20260925 04:33:54.316197 2268 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: tablet copy: Beginning tablet copy session from remote peer at address 127.27.46.254:46331
I20260925 04:33:54.326649 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.316895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.27.46.254:46331)
0925 04:33:54.317030 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.317035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.317963 (+ 928us) server_negotiation.cc:407] Connection header received
0925 04:33:54.318181 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.318184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.318265 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.318362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.319001 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.319837 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.321114 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.321465 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.321597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.323039 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.323042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.323045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.323074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.324779 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.325282 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.325286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.325288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.325374 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.325710 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.325714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.325715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.325924 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.326022 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.326364 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.326490 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:33:54.326645 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.316781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:47220)
0925 04:33:54.317752 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.317770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.318068 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.318388 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.318391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.318697 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.318860 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.318870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.319965 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.319971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.320977 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.320989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.321613 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.321616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.321727 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.322347 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.322380 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.322867 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.324913 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.324916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.324919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.325170 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.325471 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.325473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.325475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.325611 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.326028 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.326030 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.326148 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.326300 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.326449 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":103,"threads_started":1}
I20260925 04:33:54.327642 1698 tablet_copy_service.cc:140] P 6407d49569d8484182db8b9084d335b8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e75683aea8a34316b72bc2b39c65d087 ({username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:47220)
I20260925 04:33:54.327762 1698 tablet_copy_service.cc:161] P 6407d49569d8484182db8b9084d335b8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e75683aea8a34316b72bc2b39c65d087 at {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:47220: session id = e75683aea8a34316b72bc2b39c65d087-00000000000000000000000000000000
I20260925 04:33:54.329059 1698 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:33:54.330386 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:33:54.332762 2268 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: tablet copy: Starting download of 0 data blocks...
I20260925 04:33:54.332989 2268 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: tablet copy: Starting download of 1 WAL segments...
I20260925 04:33:54.334604 2268 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:33:54.336426 1698 tablet_copy_service.cc:342] P 6407d49569d8484182db8b9084d335b8: Request end of tablet copy session e75683aea8a34316b72bc2b39c65d087-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:47220
I20260925 04:33:54.336515 1698 tablet_copy_service.cc:433] P 6407d49569d8484182db8b9084d335b8: ending tablet copy session e75683aea8a34316b72bc2b39c65d087-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e75683aea8a34316b72bc2b39c65d087
I20260925 04:33:54.343995 2268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:54.345913 2458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.346563 2459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.349697 2461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.354180 2268 server_base.cc:1034] running on GCE node
I20260925 04:33:54.354422 2268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:54.354780 2268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:54.355948 2268 hybrid_clock.cc:648] HybridClock initialized: now 1790310834355934 us; error 33 us; skew 500 ppm
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:54.357834 2268 init.cc:377] Logged in from keytab as oryx/127.27.46.252@KRBTEST.COM (short username oryx)
I20260925 04:33:54.358319 2268 webserver.cc:460] Webserver started at http://127.27.46.252:39959/ using document root <none> and password file <none>
I20260925 04:33:54.358536 2268 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.358587 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.359987 2268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:54.361089 2468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.361388 2268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:54.361464 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "e75683aea8a34316b72bc2b39c65d087"
format_stamp: "Formatted at 2026-09-25 04:33:54 on dist-test-slave-r8r3"
server_key: "804e9a83919fccc8449410430c3185a4"
server_key_iv: "22a05b1144424d7bebcb9b96f726cd2e"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.361578 2268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260925 04:33:54.374161 2446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:54.374485 2446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:54.374548 2446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:33:54.376749 2268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:54.379032 2268 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260925 04:33:54.380828 2446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:54.380904 2446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:54.381057 2446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:54.384426 2268 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.252:35541
I20260925 04:33:54.384991 2268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
I20260925 04:33:54.386809 2524 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:33:54.388007 2524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Bootstrap starting.
I20260925 04:33:54.388883 2446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:44017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=33531
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:54.390281 2446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:54.391543 2446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:33:54.393628 2523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.252:35541 every 8 connection(s)
I20260925 04:33:54.400318 2524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:33:54.400722 2524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Bootstrap complete.
I20260925 04:33:54.401429 2524 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } }
I20260925 04:33:54.401690 2446 server_base.cc:1034] running on GCE node
I20260925 04:33:54.401738 2524 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e75683aea8a34316b72bc2b39c65d087, State: Initialized, Role: LEARNER
W20260925 04:33:54.401650 2529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.401922 2531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.402113 2528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.402096 2524 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } }
I20260925 04:33:54.402618 2446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:54.402568 2533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.402701 2524 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:54.402776 2533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:54.403316 2446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:54.404498 2446 hybrid_clock.cc:648] HybridClock initialized: now 1790310834404434 us; error 77 us; skew 500 ppm
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:54.407606 2545 catalog_manager.cc:1259] Loaded cluster ID: d7c2849ecbd04a16ae416b8451be5db5
I20260925 04:33:54.407684 2545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: loading cluster ID for follower catalog manager: success
I20260925 04:33:54.407859 2446 init.cc:377] Logged in from keytab as oryx/127.27.46.193@KRBTEST.COM (short username oryx)
I20260925 04:33:54.408900 2446 webserver.cc:460] Webserver started at http://127.27.46.193:33531/ using document root <none> and password file <none>
I20260925 04:33:54.409535 2545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: acquiring CA information for follower catalog manager: success
I20260925 04:33:54.409708 2446 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.409801 2446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.410729 2545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:33:54.411882 2009 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
I20260925 04:33:54.414234 2446 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:33:54.416178 2550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.417366 2446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:54.417491 2446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "3cf2875e42214818b62f43e6565e1acc"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "fdf456f0d5838e6d31332ff0282a8b49"
server_key_iv: "e844fdd29943dbdcc71a65eed3e12a40"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.418020 2446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:54.428810 2446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:54.431520 2446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:54.431712 2446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:54.432260 2446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:54.433059 2446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:54.433101 2446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.433166 2446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:54.433193 2446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.443506 2446 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44017
I20260925 04:33:54.443607 2665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44017 every 8 connection(s)
I20260925 04:33:54.444528 2446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:54.450508 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2446
I20260925 04:33:54.450729 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1876
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
I20260925 04:33:54.464347 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.446368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:32999 (local address 127.27.46.254:46331)
0925 04:33:54.446508 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.446513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.448058 (+ 1545us) server_negotiation.cc:407] Connection header received
0925 04:33:54.449053 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.449059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.449120 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.449202 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.451171 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452159 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.454466 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.454919 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.455008 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.458527 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.458531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.458534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.458574 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.461691 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462451 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.462455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.462457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.462549 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.463084 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.463090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.463095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.463299 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.463398 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.464011 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.464140 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:54.464380 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.446370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57383 (local address 127.27.46.253:37271)
0925 04:33:54.446508 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.446513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.448070 (+ 1557us) server_negotiation.cc:407] Connection header received
0925 04:33:54.449379 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.449382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.449440 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.449552 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.451173 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452026 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.454451 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.454796 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.454890 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.459779 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.459786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.459792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.459827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.461684 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462401 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.462407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.462412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.462519 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462916 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.462922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.462926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.463165 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.463238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.463996 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.464139 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:33:54.465386 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.446777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:32999)
0925 04:33:54.447920 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.447966 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.448857 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.449381 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.449388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.450031 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.451021 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.451036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452324 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.452330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.454313 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.454327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.455240 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.455247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.455504 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.456179 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.456203 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.458345 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.461921 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.461928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.461932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462291 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.462709 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.462714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462838 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.463454 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.463456 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.463722 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.464897 (+ 1175us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.465045 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":500,"spinlock_wait_cycles":124672,"thread_start_us":124,"threads_started":1}
I20260925 04:33:54.465787 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.446778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.27.46.193:57383)
0925 04:33:54.447920 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.447964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.448840 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.449570 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.449573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.450033 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.450972 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.450994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452221 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.452225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.454295 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.454309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.455072 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.455076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.455477 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.456403 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.456416 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.459616 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.461813 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.461823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.461841 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462289 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.462631 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.462641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462815 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.463317 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.463326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.463701 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.464580 (+ 879us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.464745 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":660,"spinlock_wait_cycles":13440,"thread_start_us":758,"threads_started":1}
I20260925 04:33:54.465969 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.447252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:39491 (local address 127.27.46.252:35541)
0925 04:33:54.447831 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.447836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.448362 (+ 526us) server_negotiation.cc:407] Connection header received
0925 04:33:54.449657 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.449660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.449743 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.449842 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.451166 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452279 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.454448 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.454807 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.454911 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.460021 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.460027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.460032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.460063 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.462096 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462601 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.462608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.462609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.462698 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.463040 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.463046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.463051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.463278 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.463382 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.465625 (+ 2243us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.465757 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":134,"threads_started":1}
I20260925 04:33:54.465447 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.447036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.27.46.193:39491)
0925 04:33:54.447978 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.447989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.448857 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.449878 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.449881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.450115 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.450972 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.450994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.452415 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.452419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.454310 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.454317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.455072 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.455076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.455465 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.456522 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.456540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.459795 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.462231 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.462237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462483 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.462806 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.462809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.462811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.462926 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.463432 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.463438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.463722 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.463932 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.464281 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":632,"spinlock_wait_cycles":28544,"thread_start_us":81,"threads_started":1}
I20260925 04:33:54.466939 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:43459
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:46331
--webserver_port=41527
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:33:54.467245 2668 heartbeater.cc:347] Connected to a master server at 127.27.46.253:37271
I20260925 04:33:54.467510 2668 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.467636 2667 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:54.467739 2667 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.468109 2668 heartbeater.cc:511] Master 127.27.46.253:37271 requested a full tablet report, sending...
I20260925 04:33:54.468110 2667 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:54.468694 2666 heartbeater.cc:347] Connected to a master server at 127.27.46.252:35541
I20260925 04:33:54.468797 2666 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.468958 2666 heartbeater.cc:511] Master 127.27.46.252:35541 requested a full tablet report, sending...
I20260925 04:33:54.469352 1678 ts_manager.cc:194] Re-registered known tserver with Master: 3cf2875e42214818b62f43e6565e1acc (127.27.46.193:44017)
I20260925 04:33:54.470124 2480 ts_manager.cc:194] Registered new tserver with Master: 3cf2875e42214818b62f43e6565e1acc (127.27.46.193:44017)
I20260925 04:33:54.470295 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:32999
I20260925 04:33:54.471313 2290 ts_manager.cc:194] Registered new tserver with Master: 3cf2875e42214818b62f43e6565e1acc (127.27.46.193:44017)
W20260925 04:33:54.579706 2674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:54.579952 2674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:54.580001 2674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:54.584198 2674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:54.584252 2674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:54.584309 2674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:54.589366 2674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:43459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=41527
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:54.590466 2674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:54.591477 2674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:54.601245 2682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.601256 2679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.601256 2680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.601585 2674 server_base.cc:1034] running on GCE node
I20260925 04:33:54.602044 2674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:54.602627 2674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:54.603806 2674 hybrid_clock.cc:648] HybridClock initialized: now 1790310834603776 us; error 39 us; skew 500 ppm
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:54.606899 2674 init.cc:377] Logged in from keytab as oryx/127.27.46.194@KRBTEST.COM (short username oryx)
I20260925 04:33:54.607739 2674 webserver.cc:460] Webserver started at http://127.27.46.194:41527/ using document root <none> and password file <none>
I20260925 04:33:54.608256 2674 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.608350 2674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.612046 2674 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 04:33:54.614205 2689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.615423 2674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:33:54.615540 2674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "8f035ffe30674c738868384694eceefa"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "6abdaef7a2bc1d60348bb3aa05b41454"
server_key_iv: "9471665c6a1b8987b4ea8e1209b49455"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.616060 2674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:54.625409 2147 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
I20260925 04:33:54.633416 2674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:54.636564 2674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:54.636714 2674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:54.637202 2674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:54.638099 2674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:54.638144 2674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.638183 2674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:54.638204 2674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.646970 2674 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:43459
I20260925 04:33:54.646992 2804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:43459 every 8 connection(s)
I20260925 04:33:54.647891 2674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:54.654418 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2674
I20260925 04:33:54.654618 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2015
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
I20260925 04:33:54.662564 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:33627
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:46331
--webserver_port=43809
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:33:54.664515 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.649592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60275 (local address 127.27.46.254:46331)
0925 04:33:54.649723 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.649728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.650706 (+ 978us) server_negotiation.cc:407] Connection header received
0925 04:33:54.651797 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.651800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.651854 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.651916 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.653311 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.654097 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.657304 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.657640 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.657740 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.660037 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.660041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.660043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.660071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.661673 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.662277 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.662281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.662282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.662382 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.663054 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.663060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.663064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.663304 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.663399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.664143 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.664285 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:33:54.665705 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.650119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:38837 (local address 127.27.46.253:37271)
0925 04:33:54.650231 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.650235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.651165 (+ 930us) server_negotiation.cc:407] Connection header received
0925 04:33:54.651798 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.651801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.651854 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.651906 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.653311 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.654091 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.655366 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.655728 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.655803 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.658699 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.658704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.658707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.658737 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.661331 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.662269 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.662276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.662278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.662369 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.662816 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.662822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.662824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.663085 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.663149 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.665429 (+ 2280us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.665565 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:33:54.665705 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.649585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:46307 (local address 127.27.46.252:35541)
0925 04:33:54.649720 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.649724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.651124 (+ 1400us) server_negotiation.cc:407] Connection header received
0925 04:33:54.651779 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.651782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.651846 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.651950 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.653304 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.653877 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.658348 (+ 4471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.658700 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.658767 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.661216 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.661220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.661223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.661256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.662966 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.664400 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.664403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.664405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.664511 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.664874 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.664878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.664881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.665100 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.665167 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.665450 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.665579 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:54.666718 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.650022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.27.46.194:38837)
0925 04:33:54.651083 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.651095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.651580 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.652123 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.652131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.652567 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.653166 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.653175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.654232 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.654235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.655244 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.655256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.655979 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.655985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.656176 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.656791 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.656809 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.658535 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.661681 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.661693 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.661706 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.662149 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.662495 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.662497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.662499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.662696 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.665165 (+ 2469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.665167 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.665254 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.665877 (+ 623us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.666100 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":249,"spinlock_wait_cycles":2304,"thread_start_us":635,"threads_started":1}
I20260925 04:33:54.666719 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.650022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:60275)
0925 04:33:54.650550 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.650600 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.651580 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.652124 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.652131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.652567 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.653166 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.653175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.656417 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.656422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.657178 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.657187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.657849 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.657852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.657935 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.658468 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.658481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.659889 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.661807 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.661813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.661818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.662163 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.662493 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.662498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.662501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.662936 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.663475 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.663484 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.663820 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.664915 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.665257 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":66,"spinlock_wait_cycles":156288,"thread_start_us":192,"threads_started":1}
I20260925 04:33:54.666718 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.650022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.27.46.194:46307)
0925 04:33:54.651029 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.651040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.651582 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.652122 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.652129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.652558 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.653154 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.653168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.654010 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.654013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.658221 (+ 4208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.658231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.658924 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.658929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.659033 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.659506 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.659525 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.661060 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.664019 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.664024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.664028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.664277 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.664640 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.664644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.664648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.664743 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.665224 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.665226 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.665284 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.665877 (+ 593us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.666058 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":128,"thread_start_us":152,"threads_started":1}
I20260925 04:33:54.668207 2806 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:54.668414 2805 heartbeater.cc:347] Connected to a master server at 127.27.46.252:35541
I20260925 04:33:54.668471 2806 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.668502 2805 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.669029 2807 heartbeater.cc:347] Connected to a master server at 127.27.46.253:37271
I20260925 04:33:54.668998 2805 heartbeater.cc:511] Master 127.27.46.252:35541 requested a full tablet report, sending...
I20260925 04:33:54.669132 2807 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.669302 2807 heartbeater.cc:511] Master 127.27.46.253:37271 requested a full tablet report, sending...
I20260925 04:33:54.669003 2806 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:54.670248 2480 ts_manager.cc:194] Registered new tserver with Master: 8f035ffe30674c738868384694eceefa (127.27.46.194:43459)
I20260925 04:33:54.670315 1678 ts_manager.cc:194] Re-registered known tserver with Master: 8f035ffe30674c738868384694eceefa (127.27.46.194:43459)
I20260925 04:33:54.670694 2290 ts_manager.cc:194] Registered new tserver with Master: 8f035ffe30674c738868384694eceefa (127.27.46.194:43459)
I20260925 04:33:54.671168 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:60275
W20260925 04:33:54.773978 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:54.774221 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:54.774260 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:54.778239 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:54.778286 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:54.778342 2812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:54.783551 2812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:33627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=43809
--tserver_master_addrs=127.27.46.254:46331,127.27.46.253:37271,127.27.46.252:35541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:54.784577 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:54.785481 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:33:54.795008 2812 server_base.cc:1034] running on GCE node
W20260925 04:33:54.794956 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.795007 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:54.794956 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:54.795725 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:54.796324 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:54.797569 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790310834797546 us; error 30 us; skew 500 ppm
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:54.800745 2812 init.cc:377] Logged in from keytab as oryx/127.27.46.195@KRBTEST.COM (short username oryx)
I20260925 04:33:54.801718 2812 webserver.cc:460] Webserver started at http://127.27.46.195:43809/ using document root <none> and password file <none>
I20260925 04:33:54.802218 2812 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:54.802290 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:54.806354 2812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:54.808240 2828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.809413 2812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:54.809578 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "eddd6246aa17499bb0c827a34aed67be"
format_stamp: "Formatted at 2026-09-25 04:33:53 on dist-test-slave-r8r3"
server_key: "ca43972d529af05db9d15973f9607131"
server_key_iv: "68efd80506eac6b393ac15ac128e99f0"
server_key_version: "encryptionkey@0"
I20260925 04:33:54.810079 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:54.823930 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:54.827144 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:54.827371 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:54.827960 2812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:33:54.828812 2812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:54.828869 2812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.828935 2812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:54.828970 2812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:54.838699 2812 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:33627
I20260925 04:33:54.838732 2943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:33627 every 8 connection(s)
I20260925 04:33:54.839720 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:54.850004 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 2812
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.252@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.253@KRBTEST.COM
Sep 25 04:33:54 dist-test-slave-r8r3 krb5kdc[1633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:54.856547 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.841553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:55031 (local address 127.27.46.252:35541)
0925 04:33:54.841690 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.841695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.843413 (+ 1718us) server_negotiation.cc:407] Connection header received
0925 04:33:54.843901 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.843905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.843962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.844034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.845515 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.846327 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.847673 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.847896 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848002 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.852318 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.852325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.852329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.852360 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.854184 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854782 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.854786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.854788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.854894 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855224 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.855227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.855228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.855436 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.855519 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.856216 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.856366 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:33:54.856717 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.841549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36229 (local address 127.27.46.254:46331)
0925 04:33:54.841689 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.841694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.843004 (+ 1310us) server_negotiation.cc:407] Connection header received
0925 04:33:54.843901 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.843905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.843962 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.844018 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.845517 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.846325 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848097 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.848414 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848505 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.852251 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.852255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.852257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.852283 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.854057 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854619 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.854624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.854628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.854712 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855119 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.855125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.855130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.855417 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.855826 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.856384 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.856581 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:33:54.857468 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.842996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.27.46.195:55031)
0925 04:33:54.843294 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.843306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.843647 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.844291 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.844297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.844788 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.845342 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.845350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.846434 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.846439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.847548 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.847558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.848039 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.848247 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.849056 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.849075 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.852155 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.854316 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.854324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.854668 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.855018 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.855028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.855120 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.855597 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.855600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.855919 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.856811 (+ 892us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.856961 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":136,"spinlock_wait_cycles":43520,"thread_start_us":83,"threads_started":1}
I20260925 04:33:54.857801 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.841979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.27.46.195:55357)
0925 04:33:54.842538 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.842603 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.843647 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.844288 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.844294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.844766 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.845329 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.845341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.846185 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.846188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.847551 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.847560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.848221 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.848309 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.848989 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.849006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.851572 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.854509 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.854514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.855075 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.855360 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.855365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.855462 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.855890 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.855893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.855983 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.856128 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.856524 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":130,"spinlock_wait_cycles":4608,"thread_start_us":671,"threads_started":1}
I20260925 04:33:54.858592 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.841979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:36229)
0925 04:33:54.842886 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.842898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.843656 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.844291 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.844297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.844788 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:54.845340 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.845350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.847193 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.847196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.847971 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.847979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.848662 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.848774 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.849198 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:54.849212 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:54.852122 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:54.854173 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.854192 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.854502 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.854815 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:54.854818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:54.854819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:54.855005 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:54.855548 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:54.855555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:54.855920 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.856128 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.858429 (+ 2301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":2197,"spinlock_wait_cycles":19072,"thread_start_us":123,"threads_started":1}
I20260925 04:33:54.858834 2944 heartbeater.cc:347] Connected to a master server at 127.27.46.252:35541
I20260925 04:33:54.859071 2944 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.859174 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.843058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:55357 (local address 127.27.46.253:37271)
0925 04:33:54.843171 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.843176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.843200 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:33:54.843981 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.843984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.844029 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.844122 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.845473 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.846034 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.847679 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.848004 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.848083 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.851739 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.851745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.851748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.851778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.854409 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855178 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.855182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.855183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.855258 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.855563 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.855566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.855568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.855759 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.855848 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.858895 (+ 3047us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.859021 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:33:54.859624 2944 heartbeater.cc:511] Master 127.27.46.252:35541 requested a full tablet report, sending...
I20260925 04:33:54.859711 2946 heartbeater.cc:347] Connected to a master server at 127.27.46.253:37271
I20260925 04:33:54.859797 2946 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.859964 2946 heartbeater.cc:511] Master 127.27.46.253:37271 requested a full tablet report, sending...
I20260925 04:33:54.860634 2945 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46331
I20260925 04:33:54.860704 2945 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:54.860754 2290 ts_manager.cc:194] Registered new tserver with Master: eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:54.860864 2945 heartbeater.cc:511] Master 127.27.46.254:46331 requested a full tablet report, sending...
I20260925 04:33:54.861279 2480 ts_manager.cc:194] Registered new tserver with Master: eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:54.861388 1678 ts_manager.cc:194] Re-registered known tserver with Master: eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:54.862172 1678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:36229
I20260925 04:33:54.863880 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.858867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.0.0.1:33736)
0925 04:33:54.859394 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.859409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.859500 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.859902 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.859906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.859923 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:54.860020 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.860026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.860965 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.860968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.862164 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.862172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.863395 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.863400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.863499 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.863524 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.863610 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.863704 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":65,"threads_started":1}
I20260925 04:33:54.864312 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.859137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.27.46.252:35541)
0925 04:33:54.859272 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.859277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.859585 (+ 308us) server_negotiation.cc:407] Connection header received
0925 04:33:54.859631 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.859634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.859695 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.859791 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:54.860145 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.860838 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.862335 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.863062 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.863172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.863327 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.864092 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.864168 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:33:54.864843 2495 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 04:33:54.865295 2533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6407d49569d8484182db8b9084d335b8. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.865455 2533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:33:54.866592 2956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.852451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.27.46.253:37271)
0925 04:33:54.853001 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.853006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.855961 (+ 2955us) server_negotiation.cc:407] Connection header received
0925 04:33:54.856000 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.856002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.856044 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.856119 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.857140 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.857891 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.859324 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.859660 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.859761 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.862572 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.862576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.862579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.862600 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.864442 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.864923 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.864925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.864927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.865001 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.865271 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.865275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.865276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.865526 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.865654 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.866305 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.866438 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":90,"threads_started":1}
I20260925 04:33:54.871744 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.855205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.27.46.252:35541)
0925 04:33:54.856718 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.856723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.862120 (+ 5397us) server_negotiation.cc:407] Connection header received
0925 04:33:54.862309 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.862311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.862355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.862400 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.864398 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.865119 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.866340 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.866650 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.866735 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.868421 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.868425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.868427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.868451 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.869990 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.870509 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.870513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.870514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.870598 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.870910 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.870914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.870916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.871124 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.871260 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.871471 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.871593 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1444,"thread_start_us":67,"threads_started":1}
I20260925 04:33:54.871919 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.854763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.27.46.254:46331)
0925 04:33:54.856862 (+ 2099us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.856866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.861927 (+ 5061us) server_negotiation.cc:407] Connection header received
0925 04:33:54.863307 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.863310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.863356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.863405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:54.864574 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.865212 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.866905 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.867188 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.867268 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.868573 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:54.868577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:54.868579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:54.868602 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:54.870065 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.870582 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.870586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.870589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.870672 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:54.870988 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:54.870991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:54.870993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:54.871174 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:54.871287 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.871650 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.871753 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2044,"thread_start_us":100,"threads_started":1}
I20260925 04:33:54.874925 1678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:54.877966 1678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:54.880532 2823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: NON_VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: true } } }
I20260925 04:33:54.880651 2823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.883976 2823 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: attempting to promote NON_VOTER e75683aea8a34316b72bc2b39c65d087 to VOTER
I20260925 04:33:54.884321 2823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } }
I20260925 04:33:54.885457 2308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 04:33:54.885821 2497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 LEARNER]: Refusing update from remote peer 6407d49569d8484182db8b9084d335b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 04:33:54.885898 2965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 04:33:54.886524 2964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 04:33:54.888106 2965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e75683aea8a34316b72bc2b39c65d087 (127.27.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.888551 2308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e75683aea8a34316b72bc2b39c65d087 (127.27.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.888963 2497 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e75683aea8a34316b72bc2b39c65d087 (127.27.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.889806 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.889916 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.890021 2965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.890084 2964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.890202 2964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.890084 2965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:54.890903 2973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.891019 2973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:54.891198 2533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6407d49569d8484182db8b9084d335b8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:54.891312 2533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:54.897737 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.892569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33627 (local address 127.0.0.1:47376)
0925 04:33:54.893403 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.893417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.893579 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.894049 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.894052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.894070 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:54.894163 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.894169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895315 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.895321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.896473 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.896481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.897270 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.897273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.897408 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.897432 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.897527 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.897599 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":29,"thread_start_us":86,"threads_started":1}
I20260925 04:33:54.897959 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.892714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.27.46.195:33627)
0925 04:33:54.893241 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.893252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.893546 (+ 294us) server_negotiation.cc:407] Connection header received
0925 04:33:54.893705 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.893710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.893914 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.894052 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:54.894292 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895204 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.896597 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.897145 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.897269 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.897425 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.897556 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.897649 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":86,"threads_started":1}
I20260925 04:33:54.898921 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.892264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:43459 (local address 127.0.0.1:47712)
0925 04:33:54.892422 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.892437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.892511 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.893705 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.893708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.893727 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:54.893823 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.893829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895829 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.895843 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.897378 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.897384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.898510 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.898514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.898597 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.898626 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.898725 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.898800 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":208}
I20260925 04:33:54.899088 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.892862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44017 (local address 127.0.0.1:36896)
0925 04:33:54.893695 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:54.893709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:54.893780 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:54.894288 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:54.894291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:54.894322 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:54.894417 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.894423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895828 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.895843 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.897365 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:54.897377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.898694 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.898711 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:54.898794 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.898823 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:33:54.898908 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:33:54.898971 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":137,"thread_start_us":76,"threads_started":1}
I20260925 04:33:54.899485 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.893271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.27.46.193:44017)
0925 04:33:54.893844 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.893856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.893875 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:33:54.893944 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.893947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.894128 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.894279 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:54.894558 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895690 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.897531 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.898383 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.898494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.898625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.899241 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.899330 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":122,"threads_started":1}
I20260925 04:33:54.900997 2878 tablet_service.cc:1467] Processing CreateTablet for tablet e935528243384435b48be20eb81a43cf (DEFAULT_TABLE table=test-table [id=0107e2024bb340c9ba5a2ab059ee2b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:54.901808 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:54.892688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.27.46.194:43459)
0925 04:33:54.893134 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:54.893147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:54.893170 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:33:54.893250 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:54.893256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:54.893549 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:54.893975 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:54.894116 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.895690 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.897531 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:54.898383 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:54.898494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:54.898625 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:54.901597 (+ 2972us) server_negotiation.cc:299] Negotiation successful
0925 04:33:54.901690 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":97,"threads_started":1}
I20260925 04:33:54.902091 2599 tablet_service.cc:1467] Processing CreateTablet for tablet e935528243384435b48be20eb81a43cf (DEFAULT_TABLE table=test-table [id=0107e2024bb340c9ba5a2ab059ee2b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:54.902809 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e935528243384435b48be20eb81a43cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:54.903044 2599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e935528243384435b48be20eb81a43cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:54.904234 2739 tablet_service.cc:1467] Processing CreateTablet for tablet e935528243384435b48be20eb81a43cf (DEFAULT_TABLE table=test-table [id=0107e2024bb340c9ba5a2ab059ee2b44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:54.905100 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e935528243384435b48be20eb81a43cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:54.911121 2979 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Bootstrap starting.
I20260925 04:33:54.911710 2981 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Bootstrap starting.
I20260925 04:33:54.913312 2980 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Bootstrap starting.
I20260925 04:33:54.913789 2979 tablet_bootstrap.cc:654] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:54.914938 2979 log.cc:824] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:54.915604 2980 tablet_bootstrap.cc:654] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:54.915930 2981 tablet_bootstrap.cc:654] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:54.916426 2980 log.cc:824] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:54.916635 2981 log.cc:824] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:54.918200 2979 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: No bootstrap required, opened a new log
I20260925 04:33:54.918325 2981 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: No bootstrap required, opened a new log
I20260925 04:33:54.918385 2979 ts_tablet_manager.cc:1397] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260925 04:33:54.918481 2981 ts_tablet_manager.cc:1397] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 04:33:54.923050 2981 raft_consensus.cc:348] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.923050 2979 raft_consensus.cc:348] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.923295 2979 raft_consensus.cc:374] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:54.923295 2981 raft_consensus.cc:374] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:54.923341 2979 raft_consensus.cc:729] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f035ffe30674c738868384694eceefa, State: Initialized, Role: FOLLOWER
I20260925 04:33:54.923341 2981 raft_consensus.cc:729] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf2875e42214818b62f43e6565e1acc, State: Initialized, Role: FOLLOWER
I20260925 04:33:54.923815 2981 consensus_queue.cc:260] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.923815 2979 consensus_queue.cc:260] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.924552 2806 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
I20260925 04:33:54.924847 2981 ts_tablet_manager.cc:1428] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:33:54.924849 2979 ts_tablet_manager.cc:1428] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 04:33:54.925164 2667 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
I20260925 04:33:54.926649 2980 tablet_bootstrap.cc:492] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: No bootstrap required, opened a new log
I20260925 04:33:54.926846 2980 ts_tablet_manager.cc:1397] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260925 04:33:54.931587 2980 raft_consensus.cc:348] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.931891 2980 raft_consensus.cc:374] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:54.931952 2980 raft_consensus.cc:729] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddd6246aa17499bb0c827a34aed67be, State: Initialized, Role: FOLLOWER
I20260925 04:33:54.932382 2980 consensus_queue.cc:260] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:54.933357 2980 ts_tablet_manager.cc:1428] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260925 04:33:54.933569 2945 heartbeater.cc:503] Master 127.27.46.254:46331 was elected leader, sending a full tablet report...
W20260925 04:33:54.946363 2669 tablet.cc:2369] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:33:55.092072 2947 tablet.cc:2369] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:33:55.149084 2808 tablet.cc:2369] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:55.285898 2986 raft_consensus.cc:482] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:55.286118 2986 raft_consensus.cc:504] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:55.287273 2986 leader_election.cc:288] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f035ffe30674c738868384694eceefa (127.27.46.194:43459), eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:55.291203 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.287679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:36417 (local address 127.27.46.194:43459)
0925 04:33:55.287827 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.287831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.287943 (+ 112us) server_negotiation.cc:407] Connection header received
0925 04:33:55.288083 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.288087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.288141 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.288231 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:55.288544 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.289035 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290209 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.290604 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290739 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.290792 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.290999 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.291068 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:33:55.291239 2976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.288143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57269 (local address 127.27.46.195:33627)
0925 04:33:55.288275 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.288279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.288294 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:33:55.288423 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.288425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.288464 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.288541 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:55.288800 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.289272 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290298 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.290688 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290816 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.290865 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.291043 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.291111 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:33:55.291261 2989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.287961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33627 (local address 127.27.46.193:57269)
0925 04:33:55.288189 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:55.288204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:55.288297 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:55.288573 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:55.288576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:55.288592 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:55.288686 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:55.288691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.289384 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.289388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:55.290164 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:55.290172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.290812 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:55.290919 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.290937 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:33:55.291025 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:33:55.291095 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"spinlock_wait_cycles":3712,"thread_start_us":87,"threads_started":1}
I20260925 04:33:55.291275 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.287562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:43459 (local address 127.27.46.193:36417)
0925 04:33:55.287829 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:55.287846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:55.287961 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:55.288251 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:55.288254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:55.288289 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:55.288418 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:55.288423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.289156 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.289160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:55.290077 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:55.290086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.290732 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.290736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:55.290851 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.290879 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:33:55.290980 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:33:55.291056 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"spinlock_wait_cycles":7296,"thread_start_us":106,"threads_started":1}
I20260925 04:33:55.292016 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e935528243384435b48be20eb81a43cf" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddd6246aa17499bb0c827a34aed67be" is_pre_election: true
I20260925 04:33:55.292016 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e935528243384435b48be20eb81a43cf" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f035ffe30674c738868384694eceefa" is_pre_election: true
I20260925 04:33:55.292313 2759 raft_consensus.cc:2454] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 0.
I20260925 04:33:55.292325 2898 raft_consensus.cc:2454] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 0.
I20260925 04:33:55.292871 2552 leader_election.cc:302] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf2875e42214818b62f43e6565e1acc, eddd6246aa17499bb0c827a34aed67be; no voters:
I20260925 04:33:55.293155 2986 raft_consensus.cc:2781] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:55.293234 2986 raft_consensus.cc:482] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:55.293264 2986 raft_consensus.cc:3037] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:55.294561 2986 raft_consensus.cc:504] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:55.294935 2986 leader_election.cc:288] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 election: Requested vote from peers 8f035ffe30674c738868384694eceefa (127.27.46.194:43459), eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:55.295356 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e935528243384435b48be20eb81a43cf" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f035ffe30674c738868384694eceefa"
I20260925 04:33:55.295408 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e935528243384435b48be20eb81a43cf" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddd6246aa17499bb0c827a34aed67be"
I20260925 04:33:55.295485 2759 raft_consensus.cc:3037] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:55.295528 2898 raft_consensus.cc:3037] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:55.296810 2759 raft_consensus.cc:2454] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 1.
I20260925 04:33:55.296810 2898 raft_consensus.cc:2454] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 1.
I20260925 04:33:55.297112 2552 leader_election.cc:302] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf2875e42214818b62f43e6565e1acc, eddd6246aa17499bb0c827a34aed67be; no voters:
I20260925 04:33:55.297281 2986 raft_consensus.cc:2781] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:55.297483 2986 raft_consensus.cc:686] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 1 LEADER]: Becoming Leader. State: Replica: 3cf2875e42214818b62f43e6565e1acc, State: Running, Role: LEADER
I20260925 04:33:55.297885 2986 consensus_queue.cc:237] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } }
I20260925 04:33:55.301637 1678 catalog_manager.cc:5495] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf2875e42214818b62f43e6565e1acc (127.27.46.193). New cstate: current_term: 1 leader_uuid: "3cf2875e42214818b62f43e6565e1acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } health_report { overall_health: HEALTHY } } }
I20260925 04:33:55.349568 2978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.345613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.27.46.193:44017)
0925 04:33:55.345756 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.345760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.345774 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:55.345944 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.345947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.346004 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.346113 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:55.346486 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.347010 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.347842 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.348080 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.348200 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.348641 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:55.349084 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:55.349220 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.349275 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.349334 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:33:55.355880 2759 raft_consensus.cc:1261] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Refusing update from remote peer 3cf2875e42214818b62f43e6565e1acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:55.355933 2898 raft_consensus.cc:1261] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Refusing update from remote peer 3cf2875e42214818b62f43e6565e1acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:55.356655 2986 consensus_queue.cc:1035] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:55.356873 2990 consensus_queue.cc:1035] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:55.367197 2995 mvcc.cc:205] Tried to move back new op lower bound from 7333113181609701376 to 7333113181382725632. Current Snapshot: MvccSnapshot[applied={T|T < 7333113181609701376}]
I20260925 04:33:55.376145 1677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:55.376336 1677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:55.380697 1677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6407d49569d8484182db8b9084d335b8: Sending DeleteTablet for 3 replicas of tablet e935528243384435b48be20eb81a43cf
I20260925 04:33:55.381807 2878 tablet_service.cc:1514] Processing DeleteTablet for tablet e935528243384435b48be20eb81a43cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:55 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47376
I20260925 04:33:55.382606 3004 tablet_replica.cc:330] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: stopping tablet replica
I20260925 04:33:55.382704 2599 tablet_service.cc:1514] Processing DeleteTablet for tablet e935528243384435b48be20eb81a43cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:55 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:36896
I20260925 04:33:55.382957 3004 raft_consensus.cc:2229] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:55.383266 3005 tablet_replica.cc:330] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: stopping tablet replica
I20260925 04:33:55.383294 3004 raft_consensus.cc:2258] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:55.383423 2739 tablet_service.cc:1514] Processing DeleteTablet for tablet e935528243384435b48be20eb81a43cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:55 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47712
I20260925 04:33:55.384142 3005 raft_consensus.cc:2229] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:55.384712 3005 raft_consensus.cc:2258] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:55.385142 3004 ts_tablet_manager.cc:1905] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:55.387524 3005 ts_tablet_manager.cc:1905] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:55.388375 3004 ts_tablet_manager.cc:1918] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:55.388469 3004 log.cc:1196] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/e935528243384435b48be20eb81a43cf
I20260925 04:33:55.388777 3004 ts_tablet_manager.cc:1939] T e935528243384435b48be20eb81a43cf P eddd6246aa17499bb0c827a34aed67be: Deleting consensus metadata
I20260925 04:33:55.390029 1663 catalog_manager.cc:4841] TS eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627): tablet e935528243384435b48be20eb81a43cf (table test-table [id=0107e2024bb340c9ba5a2ab059ee2b44]) successfully deleted
I20260925 04:33:55.391611 3005 ts_tablet_manager.cc:1918] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:55.391655 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 1646
I20260925 04:33:55.392836 3005 log.cc:1196] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/e935528243384435b48be20eb81a43cf
I20260925 04:33:55.393576 3005 ts_tablet_manager.cc:1939] T e935528243384435b48be20eb81a43cf P 3cf2875e42214818b62f43e6565e1acc: Deleting consensus metadata
I20260925 04:33:55.403878 3006 tablet_replica.cc:330] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: stopping tablet replica
I20260925 04:33:55.404275 3006 raft_consensus.cc:2229] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:55.404745 3006 raft_consensus.cc:2258] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:33:55.405230 2551 connection.cc:535] server connection from 127.0.0.1:36896 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 04:33:55.407021 2471 connection.cc:535] server connection from 127.0.0.1:33736 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:33:55.408911 3006 ts_tablet_manager.cc:1905] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:55.412528 3006 ts_tablet_manager.cc:1918] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:55.412654 3006 log.cc:1196] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/e935528243384435b48be20eb81a43cf
I20260925 04:33:55.413102 3006 ts_tablet_manager.cc:1939] T e935528243384435b48be20eb81a43cf P 8f035ffe30674c738868384694eceefa: Deleting consensus metadata
W20260925 04:33:55.414532 2691 connection.cc:441] server connection from 127.0.0.1:47712 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47712 (request call id 1) could send its response
I20260925 04:33:55.419226 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.387654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.27.46.253:37271)
0925 04:33:55.390415 (+ 2761us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.390436 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.402166 (+ 11730us) server_negotiation.cc:407] Connection header received
0925 04:33:55.402386 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.402391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.402458 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.402526 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:55.405070 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.405989 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.407393 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.407738 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.407812 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.414289 (+ 6477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:55.414293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:55.414296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:55.414328 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:55.416706 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.417463 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.417467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.417469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.417590 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.418003 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.418010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.418011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.418244 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:55.418414 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.418835 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.418981 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2657,"thread_start_us":131,"threads_started":1}
I20260925 04:33:55.430891 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.401677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.27.46.252:35541)
0925 04:33:55.407318 (+ 5641us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.407322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.420671 (+ 13349us) server_negotiation.cc:407] Connection header received
0925 04:33:55.420839 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.420856 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.420900 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.420955 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:55.421763 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.422564 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.423703 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.424057 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.424126 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.426666 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:55.426670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:55.426673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:55.426724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:55.428918 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.429363 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.429366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.429368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.429455 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.429921 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.429925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.429927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.430129 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:55.430241 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.430566 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.430700 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5559,"thread_start_us":64,"threads_started":1}
I20260925 04:33:55.432210 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.414876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.27.46.253:37271)
0925 04:33:55.415572 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.415579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.416362 (+ 783us) server_negotiation.cc:407] Connection header received
0925 04:33:55.416587 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.416591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.416646 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.416736 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:55.417657 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.418506 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.419758 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.420203 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.420329 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.423712 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:55.423717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:55.423720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:55.423751 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:55.427184 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.428458 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.428462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.428464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.428585 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.430083 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.430087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.430090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.430348 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:55.430495 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.431861 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.431990 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":62,"threads_started":1}
I20260925 04:33:55.434808 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:55.415531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.27.46.252:35541)
0925 04:33:55.415822 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:55.415828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:55.420436 (+ 4608us) server_negotiation.cc:407] Connection header received
0925 04:33:55.420496 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:55.420501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:55.420598 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:55.420703 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:55.424716 (+ 4013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.425600 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.426987 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:55.427389 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:55.427484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:55.430333 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:55.430338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:55.430340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:55.430366 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:55.432244 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.433215 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.433219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.433221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.433344 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:55.433678 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:55.433681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:55.433683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:55.433882 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:55.433986 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:55.434405 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 04:33:55.434634 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260925 04:33:56.369536 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.368990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:37609)
0925 04:33:56.369287 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.369395 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
W20260925 04:33:56.369871 2945 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.371773 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.371428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:39291)
0925 04:33:56.371626 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.371694 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
W20260925 04:33:56.372040 2806 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.373692 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.373250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:51841)
0925 04:33:56.373486 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.373600 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
W20260925 04:33:56.373925 2667 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46331 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.882364 3022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6407d49569d8484182db8b9084d335b8)
I20260925 04:33:56.882601 3022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } }
I20260925 04:33:56.883788 3022 leader_election.cc:288] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331), e75683aea8a34316b72bc2b39c65d087 (127.27.46.252:35541)
I20260925 04:33:56.884459 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.883968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36230)
0925 04:33:56.884231 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.884321 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
W20260925 04:33:56.884660 2285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.885610 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.885324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36232)
0925 04:33:56.885436 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.885546 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 04:33:56.885746 2285 leader_election.cc:334] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331): Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.887794 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.884230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:35541 (local address 127.0.0.1:57876)
0925 04:33:56.884498 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.884512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:56.884625 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:56.884949 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:56.884952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:56.884972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:56.885083 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.885088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.886094 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.886097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.886922 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.886932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.887502 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.887504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.887569 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.887590 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:33:56.887645 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 04:33:56.887712 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
I20260925 04:33:56.887862 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.884209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.27.46.252:35541)
0925 04:33:56.884498 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:56.884504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:56.884618 (+ 114us) server_negotiation.cc:407] Connection header received
0925 04:33:56.884743 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:56.884745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:56.884823 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:56.884913 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:56.885212 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.885974 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.887032 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.887410 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.887486 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.887539 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:56.887686 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:33:56.887736 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260925 04:33:56.888304 2497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a455d41894b401b8ccc392de5e2b3fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e75683aea8a34316b72bc2b39c65d087" is_pre_election: true
I20260925 04:33:56.888842 2283 leader_election.cc:302] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a455d41894b401b8ccc392de5e2b3fa; no voters: 6407d49569d8484182db8b9084d335b8, e75683aea8a34316b72bc2b39c65d087
I20260925 04:33:56.889091 3022 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:33:56.891728 3027 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6407d49569d8484182db8b9084d335b8)
I20260925 04:33:56.891876 3027 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } }
I20260925 04:33:56.892778 3027 leader_election.cc:288] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331), 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253:37271)
I20260925 04:33:56.893253 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.892818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36246)
0925 04:33:56.893074 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.893146 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":60,"threads_started":1}
W20260925 04:33:56.893427 2472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.894335 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.894100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36258)
0925 04:33:56.894209 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.894272 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:33:56.894487 2472 leader_election.cc:334] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331): Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.896692 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.892962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:37271 (local address 127.0.0.1:53026)
0925 04:33:56.893188 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.893203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:56.893293 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:56.894077 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:56.894081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:56.894101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:56.894195 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.894202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.895041 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.895044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.895829 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.895837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.896411 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.896413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.896474 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.896492 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:33:56.896558 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:33:56.896607 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260925 04:33:56.896721 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.893036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.27.46.253:37271)
0925 04:33:56.893365 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:56.893377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:56.893393 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:33:56.893895 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:56.893899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:56.893950 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:56.894052 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:56.894313 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.894944 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.895933 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.896316 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.896400 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.896446 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:56.896587 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 04:33:56.896639 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260925 04:33:56.897119 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e75683aea8a34316b72bc2b39c65d087" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a455d41894b401b8ccc392de5e2b3fa" is_pre_election: true
I20260925 04:33:56.897281 2308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75683aea8a34316b72bc2b39c65d087 in term 1.
I20260925 04:33:56.897701 2471 leader_election.cc:302] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a455d41894b401b8ccc392de5e2b3fa, e75683aea8a34316b72bc2b39c65d087; no voters: 6407d49569d8484182db8b9084d335b8
I20260925 04:33:56.897882 3027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:33:56.897938 3027 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6407d49569d8484182db8b9084d335b8)
I20260925 04:33:56.897975 3027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:33:56.898967 3027 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } }
I20260925 04:33:56.899305 3027 leader_election.cc:288] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 election: Requested vote from peers 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331), 3a455d41894b401b8ccc392de5e2b3fa (127.27.46.253:37271)
I20260925 04:33:56.899619 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.899371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36260)
0925 04:33:56.899504 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.899542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 04:33:56.899636 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e75683aea8a34316b72bc2b39c65d087" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a455d41894b401b8ccc392de5e2b3fa"
I20260925 04:33:56.899777 2308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:33:56.900277 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.900098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36276)
0925 04:33:56.900193 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.900235 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 04:33:56.900378 2472 leader_election.cc:334] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331): Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
I20260925 04:33:56.900923 2308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e75683aea8a34316b72bc2b39c65d087 in term 2.
I20260925 04:33:56.901192 2471 leader_election.cc:302] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3a455d41894b401b8ccc392de5e2b3fa, e75683aea8a34316b72bc2b39c65d087; no voters: 6407d49569d8484182db8b9084d335b8
I20260925 04:33:56.901389 3027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:33:56.901638 3027 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [term 2 LEADER]: Becoming Leader. State: Replica: e75683aea8a34316b72bc2b39c65d087, State: Running, Role: LEADER
I20260925 04:33:56.901886 3027 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } }
I20260925 04:33:56.903209 3032 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75683aea8a34316b72bc2b39c65d087. Latest consensus state: current_term: 2 leader_uuid: "e75683aea8a34316b72bc2b39c65d087" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:56.903357 3032 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:56.903610 3034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:56.905890 3034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:56.906281 3034 catalog_manager.cc:1259] Loaded cluster ID: d7c2849ecbd04a16ae416b8451be5db5
I20260925 04:33:56.906349 3034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:56.906908 3034 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:56.907290 3034 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Loaded TSK: 0
I20260925 04:33:56.908000 3034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:33:56.947589 2480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:56.950837 2480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:56.953966 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.953704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36292)
0925 04:33:56.953824 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.953885 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 04:33:56.954013 2308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [term 2 FOLLOWER]: Refusing update from remote peer e75683aea8a34316b72bc2b39c65d087: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 04:33:56.954627 3027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 04:33:56.954708 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.954490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.0.0.1:36308)
0925 04:33:56.954600 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.954655 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 04:33:56.954833 2472 consensus_peers.cc:487] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 -> Peer 6407d49569d8484182db8b9084d335b8 (127.27.46.254:46331): Couldn't send request to peer 6407d49569d8484182db8b9084d335b8. Status: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:33:56.956171 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75683aea8a34316b72bc2b39c65d087. Latest consensus state: current_term: 2 leader_uuid: "e75683aea8a34316b72bc2b39c65d087" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:56.956326 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:56.957481 3032 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e75683aea8a34316b72bc2b39c65d087" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:56.957664 3032 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:56.958247 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e75683aea8a34316b72bc2b39c65d087" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6407d49569d8484182db8b9084d335b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46331 } } peers { permanent_uuid: "3a455d41894b401b8ccc392de5e2b3fa" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 37271 } attrs { promote: false } } peers { permanent_uuid: "e75683aea8a34316b72bc2b39c65d087" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 35541 } attrs { promote: false } } }
I20260925 04:33:56.958369 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455d41894b401b8ccc392de5e2b3fa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:33:56.968659 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.963568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:43459 (local address 127.0.0.1:42878)
0925 04:33:56.963718 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.963735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:56.963814 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:56.965563 (+ 1749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:56.965566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:56.965583 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:56.965663 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.965669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966629 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.966632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.967321 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.967329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.968196 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.968202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.968291 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.968317 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:33:56.968416 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:33:56.968510 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:33:56.968866 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.964059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33627 (local address 127.0.0.1:59290)
0925 04:33:56.964655 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.964672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:56.964764 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:56.965022 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:56.965025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:56.965045 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:56.965179 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.965186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966265 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.966268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.967475 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.967485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.968396 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.968400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.968489 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.968515 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:33:56.968587 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:33:56.968664 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":37,"thread_start_us":100,"threads_started":1}
I20260925 04:33:56.969134 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.963718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.27.46.194:43459)
0925 04:33:56.965280 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:56.965284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:56.965298 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:56.965349 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:56.965352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:56.965401 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:56.965773 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:56.965884 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966522 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.967447 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.968081 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.968192 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.968269 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:56.968872 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 04:33:56.968979 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":86,"threads_started":1}
I20260925 04:33:56.969163 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.964177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.27.46.195:33627)
0925 04:33:56.964484 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:56.964489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:56.964738 (+ 249us) server_negotiation.cc:407] Connection header received
0925 04:33:56.964849 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:56.964852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:56.964900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:56.965292 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:56.965406 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966157 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.967600 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.968225 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.968395 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.968453 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:56.968726 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:33:56.968939 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260925 04:33:56.969748 2739 tablet_service.cc:1467] Processing CreateTablet for tablet 74faedee00c045839542a39558aec3d9 (DEFAULT_TABLE table=test-table [id=ec1486afa6bd4cedae3f7b16dbf26b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:56.969770 2878 tablet_service.cc:1467] Processing CreateTablet for tablet 74faedee00c045839542a39558aec3d9 (DEFAULT_TABLE table=test-table [id=ec1486afa6bd4cedae3f7b16dbf26b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:56.969858 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.963832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44017 (local address 127.0.0.1:40610)
0925 04:33:56.964429 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.964441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:56.964553 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:56.964930 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:56.964934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:56.964950 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:56.965047 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.965053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966132 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.966136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.967475 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:56.967485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.969410 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.969413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:56.969540 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.969562 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:33:56.969652 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:33:56.969723 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":75}
I20260925 04:33:56.970085 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74faedee00c045839542a39558aec3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:56.970086 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74faedee00c045839542a39558aec3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:56.970217 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.964440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.27.46.193:44017)
0925 04:33:56.964691 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:56.964695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:56.964710 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:33:56.964767 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:56.964769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:56.964808 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:56.964920 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:56.965204 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.966011 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.968528 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:56.969178 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:56.969288 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:56.969352 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:56.969996 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 04:33:56.970094 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
I20260925 04:33:56.972595 3045 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Bootstrap starting.
I20260925 04:33:56.972615 3046 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Bootstrap starting.
I20260925 04:33:56.973448 3046 tablet_bootstrap.cc:654] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:56.973562 3045 tablet_bootstrap.cc:654] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:56.973683 2599 tablet_service.cc:1467] Processing CreateTablet for tablet 74faedee00c045839542a39558aec3d9 (DEFAULT_TABLE table=test-table [id=ec1486afa6bd4cedae3f7b16dbf26b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:56.973981 2599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74faedee00c045839542a39558aec3d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:56.974647 3046 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: No bootstrap required, opened a new log
I20260925 04:33:56.974721 3046 ts_tablet_manager.cc:1397] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:56.975136 3046 raft_consensus.cc:348] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.975231 3046 raft_consensus.cc:374] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:56.975252 3046 raft_consensus.cc:729] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eddd6246aa17499bb0c827a34aed67be, State: Initialized, Role: FOLLOWER
I20260925 04:33:56.975591 3046 consensus_queue.cc:260] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.975971 3046 ts_tablet_manager.cc:1428] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:56.976763 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.976359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:48603)
0925 04:33:56.976606 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.976667 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260925 04:33:56.976912 3050 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc: Bootstrap starting.
I20260925 04:33:56.978165 3050 tablet_bootstrap.cc:654] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:56.978716 2944 heartbeater.cc:503] Master 127.27.46.252:35541 was elected leader, sending a full tablet report...
I20260925 04:33:56.978732 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.977757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:60701)
0925 04:33:56.978002 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.978086 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260925 04:33:56.979477 3050 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc: No bootstrap required, opened a new log
I20260925 04:33:56.979579 3045 tablet_bootstrap.cc:492] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: No bootstrap required, opened a new log
I20260925 04:33:56.979592 3050 ts_tablet_manager.cc:1397] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:56.979698 3045 ts_tablet_manager.cc:1397] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260925 04:33:56.980082 3050 raft_consensus.cc:348] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.980207 3050 raft_consensus.cc:374] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:56.980232 3050 raft_consensus.cc:729] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf2875e42214818b62f43e6565e1acc, State: Initialized, Role: FOLLOWER
I20260925 04:33:56.980247 3045 raft_consensus.cc:348] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.980370 3045 raft_consensus.cc:374] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:56.980410 3045 raft_consensus.cc:729] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f035ffe30674c738868384694eceefa, State: Initialized, Role: FOLLOWER
I20260925 04:33:56.980540 3045 consensus_queue.cc:260] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.980391 3050 consensus_queue.cc:260] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:56.980926 3045 ts_tablet_manager.cc:1428] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:33:56.980999 3050 ts_tablet_manager.cc:1428] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 04:33:56.982482 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.982025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:43307)
0925 04:33:56.982316 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.982389 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260925 04:33:56.982695 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.982265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:38889)
0925 04:33:56.982546 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.982597 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260925 04:33:56.983337 2805 heartbeater.cc:503] Master 127.27.46.252:35541 was elected leader, sending a full tablet report...
I20260925 04:33:56.983603 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.983313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:50913)
0925 04:33:56.983468 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.983518 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 04:33:56.984082 2666 heartbeater.cc:503] Master 127.27.46.252:35541 was elected leader, sending a full tablet report...
I20260925 04:33:56.985424 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:56.985186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:37561)
0925 04:33:56.985309 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:56.985359 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:33:57.087455 3055 raft_consensus.cc:482] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:57.087638 3055 raft_consensus.cc:504] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:57.088101 3055 leader_election.cc:288] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f035ffe30674c738868384694eceefa (127.27.46.194:43459), eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:57.088683 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74faedee00c045839542a39558aec3d9" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f035ffe30674c738868384694eceefa" is_pre_election: true
I20260925 04:33:57.088745 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74faedee00c045839542a39558aec3d9" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddd6246aa17499bb0c827a34aed67be" is_pre_election: true
I20260925 04:33:57.088824 2759 raft_consensus.cc:2454] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 0.
I20260925 04:33:57.088980 2898 raft_consensus.cc:2454] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 0.
I20260925 04:33:57.089215 2552 leader_election.cc:302] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf2875e42214818b62f43e6565e1acc, 8f035ffe30674c738868384694eceefa; no voters:
I20260925 04:33:57.089424 3055 raft_consensus.cc:2781] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:57.089488 3055 raft_consensus.cc:482] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:57.089583 3055 raft_consensus.cc:3037] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:57.090642 3055 raft_consensus.cc:504] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:57.090996 3055 leader_election.cc:288] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 election: Requested vote from peers 8f035ffe30674c738868384694eceefa (127.27.46.194:43459), eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627)
I20260925 04:33:57.091425 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74faedee00c045839542a39558aec3d9" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eddd6246aa17499bb0c827a34aed67be"
I20260925 04:33:57.091419 2759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74faedee00c045839542a39558aec3d9" candidate_uuid: "3cf2875e42214818b62f43e6565e1acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f035ffe30674c738868384694eceefa"
I20260925 04:33:57.091559 2898 raft_consensus.cc:3037] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:57.091620 2759 raft_consensus.cc:3037] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:57.092764 2898 raft_consensus.cc:2454] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 1.
I20260925 04:33:57.092764 2759 raft_consensus.cc:2454] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf2875e42214818b62f43e6565e1acc in term 1.
I20260925 04:33:57.093083 2552 leader_election.cc:302] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf2875e42214818b62f43e6565e1acc, eddd6246aa17499bb0c827a34aed67be; no voters:
I20260925 04:33:57.093271 3055 raft_consensus.cc:2781] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:57.093560 3055 raft_consensus.cc:686] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [term 1 LEADER]: Becoming Leader. State: Replica: 3cf2875e42214818b62f43e6565e1acc, State: Running, Role: LEADER
I20260925 04:33:57.093739 3055 consensus_queue.cc:237] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } }
I20260925 04:33:57.094347 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.094083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:44295)
0925 04:33:57.094199 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.094262 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 04:33:57.095808 2480 catalog_manager.cc:5495] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf2875e42214818b62f43e6565e1acc (127.27.46.193). New cstate: current_term: 1 leader_uuid: "3cf2875e42214818b62f43e6565e1acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf2875e42214818b62f43e6565e1acc" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:57.125573 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.122362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.27.46.193:44017)
0925 04:33:57.122521 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:57.122526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:57.122541 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:33:57.122682 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:57.122685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:57.122737 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:57.122815 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:57.123092 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:57.123620 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:57.124471 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:57.124698 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:57.124808 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:57.125076 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:57.125148 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:57.125225 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:57.125323 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 04:33:57.125381 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:57.127934 2898 raft_consensus.cc:1261] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Refusing update from remote peer 3cf2875e42214818b62f43e6565e1acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:57.127980 2759 raft_consensus.cc:1261] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Refusing update from remote peer 3cf2875e42214818b62f43e6565e1acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:57.128433 3058 consensus_queue.cc:1035] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eddd6246aa17499bb0c827a34aed67be" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:57.128597 3055 consensus_queue.cc:1035] T 74faedee00c045839542a39558aec3d9 P 3cf2875e42214818b62f43e6565e1acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f035ffe30674c738868384694eceefa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:57.130115 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.129853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:45763)
0925 04:33:57.129975 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.130038 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 04:33:57.132436 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.130773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:59157)
0925 04:33:57.132279 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.132342 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1433}
I20260925 04:33:57.134181 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.133892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.193:51649)
0925 04:33:57.134049 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.134108 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 04:33:57.134200 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.133928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.195:55471)
0925 04:33:57.134068 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.134120 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 04:33:57.134894 3062 mvcc.cc:205] Tried to move back new op lower bound from 7333113188871127040 to 7333113188737695744. Current Snapshot: MvccSnapshot[applied={T|T < 7333113188871127040}]
I20260925 04:33:57.137856 2480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:57.138037 2480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:57.138805 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.138511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46331 (local address 127.27.46.194:39871)
0925 04:33:57.138669 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.138728 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 04:33:57.142035 2480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e75683aea8a34316b72bc2b39c65d087: Sending DeleteTablet for 3 replicas of tablet 74faedee00c045839542a39558aec3d9
I20260925 04:33:57.142657 2739 tablet_service.cc:1514] Processing DeleteTablet for tablet 74faedee00c045839542a39558aec3d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:57 UTC) from {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:42878
I20260925 04:33:57.142711 2878 tablet_service.cc:1514] Processing DeleteTablet for tablet 74faedee00c045839542a39558aec3d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:57 UTC) from {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:59290
I20260925 04:33:57.142927 2738 tablet_service.cc:1514] Processing DeleteTablet for tablet 74faedee00c045839542a39558aec3d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:57 UTC) from {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:42878
I20260925 04:33:57.143097 3070 tablet_replica.cc:330] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: stopping tablet replica
I20260925 04:33:57.143134 2599 tablet_service.cc:1514] Processing DeleteTablet for tablet 74faedee00c045839542a39558aec3d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:57 UTC) from {username='oryx', principal='oryx/127.27.46.252@KRBTEST.COM'} at 127.0.0.1:40610
I20260925 04:33:57.143253 3070 raft_consensus.cc:2229] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:57.143373 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2446
I20260925 04:33:57.143415 3070 raft_consensus.cc:2258] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:57.145757 3070 ts_tablet_manager.cc:1905] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:57.147761 3070 ts_tablet_manager.cc:1918] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:57.147825 3070 log.cc:1196] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/74faedee00c045839542a39558aec3d9
I20260925 04:33:57.148089 3070 ts_tablet_manager.cc:1939] T 74faedee00c045839542a39558aec3d9 P 8f035ffe30674c738868384694eceefa: Deleting consensus metadata
I20260925 04:33:57.148911 2470 catalog_manager.cc:4841] TS 8f035ffe30674c738868384694eceefa (127.27.46.194:43459): tablet 74faedee00c045839542a39558aec3d9 (table test-table [id=ec1486afa6bd4cedae3f7b16dbf26b58]) successfully deleted
W20260925 04:33:57.149219 2470 catalog_manager.cc:4808] TS 8f035ffe30674c738868384694eceefa (127.27.46.194:43459): delete failed for tablet 74faedee00c045839542a39558aec3d9 because the tablet was not found. No further retry: Not found: Tablet not found: 74faedee00c045839542a39558aec3d9
I20260925 04:33:57.151819 3071 tablet_replica.cc:330] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: stopping tablet replica
I20260925 04:33:57.151976 3071 raft_consensus.cc:2229] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:57.152153 3071 raft_consensus.cc:2258] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:57.152307 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:57.152042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44017 (local address 127.0.0.1:40626)
0925 04:33:57.152171 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:57.152216 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:33:57.152482 2470 catalog_manager.cc:4584] TS 3cf2875e42214818b62f43e6565e1acc (127.27.46.193:44017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74faedee00c045839542a39558aec3d9: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44017: connect: Connection refused (error 111)
I20260925 04:33:57.152598 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2674
I20260925 04:33:57.152977 3071 ts_tablet_manager.cc:1905] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:57.155652 3071 ts_tablet_manager.cc:1918] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:57.155742 3071 log.cc:1196] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/74faedee00c045839542a39558aec3d9
I20260925 04:33:57.156018 3071 ts_tablet_manager.cc:1939] T 74faedee00c045839542a39558aec3d9 P eddd6246aa17499bb0c827a34aed67be: Deleting consensus metadata
I20260925 04:33:57.156689 2470 catalog_manager.cc:4841] TS eddd6246aa17499bb0c827a34aed67be (127.27.46.195:33627): tablet 74faedee00c045839542a39558aec3d9 (table test-table [id=ec1486afa6bd4cedae3f7b16dbf26b58]) successfully deleted
I20260925 04:33:57.159713 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2812
I20260925 04:33:57.165890 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2152
I20260925 04:33:57.172130 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 2268
2026-09-25T04:33:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5633 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:33:57 dist-test-slave-r8r3 krb5kdc[3075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:33:57 dist-test-slave-r8r3 krb5kdc[3075](info): set up 2 sockets
Sep 25 04:33:57 dist-test-slave-r8r3 krb5kdc[3075](info): commencing operation
krb5kdc: starting...
W20260925 04:33:58.246306 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:33:58 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:33:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab.
Entry for principal oryx/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab.
I20260925 04:33:58.382038 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41867
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab
--principal=oryx/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:58.494740 3088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:58.495020 3088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:58.495061 3088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:58.499118 3088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:33:58.499172 3088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:33:58.499190 3088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:58.499207 3088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:58.499248 3088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:33:58.499279 3088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:33:58.504493 3088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:41867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.254.keytab
--principal=oryx/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:58.505637 3088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:58.506675 3088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:58.514892 3093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:58.515081 3096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:58.515000 3088 server_base.cc:1034] running on GCE node
W20260925 04:33:58.514956 3094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:58.515753 3088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:58.516829 3088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:58.518005 3088 hybrid_clock.cc:648] HybridClock initialized: now 1790310838517976 us; error 41 us; skew 500 ppm
Sep 25 04:33:58 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:58.520582 3088 init.cc:377] Logged in from keytab as oryx/127.27.46.254@KRBTEST.COM (short username oryx)
I20260925 04:33:58.521466 3088 webserver.cc:460] Webserver started at http://127.27.46.254:33671/ using document root <none> and password file <none>
I20260925 04:33:58.522078 3088 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:58.522166 3088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:58.522389 3088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:58.524416 3088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "f1e5251a5a6d4341a776fe542f948133"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "6b87a5b32849be8042c87e343ad798d6"
server_key_iv: "7bbe668962a64a611939228560a9fa17"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.524976 3088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "f1e5251a5a6d4341a776fe542f948133"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "6b87a5b32849be8042c87e343ad798d6"
server_key_iv: "7bbe668962a64a611939228560a9fa17"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.528699 3088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:58.531071 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:58.532219 3088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:58.532357 3088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "f1e5251a5a6d4341a776fe542f948133"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "6b87a5b32849be8042c87e343ad798d6"
server_key_iv: "7bbe668962a64a611939228560a9fa17"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.532464 3088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:58.551328 3088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:58.554677 3088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:58.554885 3088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:58.562647 3088 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:41867
I20260925 04:33:58.562711 3170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:41867 every 8 connection(s)
I20260925 04:33:58.563632 3088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:33:58.566470 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:58.568753 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3088
I20260925 04:33:58.568902 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:33:58.569193 27835 external_mini_cluster.cc:1428] Setting key 41ad8f99026394aa68e2541e10fdb2fc
I20260925 04:33:58.572680 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Bootstrap starting.
I20260925 04:33:58.575120 3171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:58.575870 3171 log.cc:824] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:58.577901 3171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: No bootstrap required, opened a new log
Sep 25 04:33:58 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:58.582331 3171 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } }
I20260925 04:33:58.582602 3171 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:58.582660 3171 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e5251a5a6d4341a776fe542f948133, State: Initialized, Role: FOLLOWER
I20260925 04:33:58.583179 3171 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } }
I20260925 04:33:58.583356 3171 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:33:58.583423 3171 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:33:58.583524 3171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:58.584545 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:58.570741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.27.46.254:41867)
0925 04:33:58.571186 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:58.571201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:58.571238 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:33:58.571886 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:58.571903 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:58.572238 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:58.572591 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:58.573401 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.574502 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.575531 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.576060 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.576228 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:58.578867 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:58.578873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:58.578887 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:58.578914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:58.581334 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.581818 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.581824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.581830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.581975 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.582217 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.582220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.582222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.582753 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:58.582962 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:58.583215 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 04:33:58.583545 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260925 04:33:58.584669 3171 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } }
I20260925 04:33:58.585029 3171 leader_election.cc:302] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e5251a5a6d4341a776fe542f948133; no voters:
I20260925 04:33:58.585355 3171 leader_election.cc:288] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:33:58.585531 3176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:58.585798 3176 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [term 1 LEADER]: Becoming Leader. State: Replica: f1e5251a5a6d4341a776fe542f948133, State: Running, Role: LEADER
I20260925 04:33:58.586122 3176 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } }
I20260925 04:33:58.586562 3171 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:33:58.587409 3176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e5251a5a6d4341a776fe542f948133. Latest consensus state: current_term: 1 leader_uuid: "f1e5251a5a6d4341a776fe542f948133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } } }
I20260925 04:33:58.587535 3176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:58.587877 3178 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e5251a5a6d4341a776fe542f948133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e5251a5a6d4341a776fe542f948133" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41867 } } }
I20260925 04:33:58.587983 3178 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133 [sys.catalog]: This master's current role is: LEADER
I20260925 04:33:58.588439 3186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:33:58.591698 3186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:33:58.597697 3186 catalog_manager.cc:1347] Generated new cluster ID: 36bbc70f36e9400ca871d9f8b5e58a5c
I20260925 04:33:58.597787 3186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:33:58.608872 3186 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:33:58.609936 3186 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:33:58.642866 3186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Generated new TSK 0
I20260925 04:33:58.643884 3186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
I20260925 04:33:58.680511 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41867
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:58.793587 3198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:58.793854 3198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:58.793926 3198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:58.798029 3198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:58.798085 3198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:58.798151 3198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:58.798219 3198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:33:58.803517 3198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:58.804591 3198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:58.805682 3198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:33:58.815263 3198 server_base.cc:1034] running on GCE node
W20260925 04:33:58.815184 3203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:58.815201 3205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:58.815234 3207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:58.815912 3198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:58.816581 3198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:58.817770 3198 hybrid_clock.cc:648] HybridClock initialized: now 1790310838817741 us; error 47 us; skew 500 ppm
Sep 25 04:33:58 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:58.820466 3198 init.cc:377] Logged in from keytab as oryx/127.27.46.193@KRBTEST.COM (short username oryx)
I20260925 04:33:58.821370 3198 webserver.cc:460] Webserver started at http://127.27.46.193:43529/ using document root <none> and password file <none>
I20260925 04:33:58.821975 3198 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:58.822065 3198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:58.822285 3198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:58.824322 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "b358d43cc03b9d435d93b1ca36485547"
server_key_iv: "0e28658f579f89ec465bc6a94d84b5d1"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.824891 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "b358d43cc03b9d435d93b1ca36485547"
server_key_iv: "0e28658f579f89ec465bc6a94d84b5d1"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.828560 3198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:33:58.830852 3214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:58.832036 3198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:58.832161 3198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
format_stamp: "Formatted at 2026-09-25 04:33:58 on dist-test-slave-r8r3"
server_key: "b358d43cc03b9d435d93b1ca36485547"
server_key_iv: "0e28658f579f89ec465bc6a94d84b5d1"
server_key_version: "encryptionkey@0"
I20260925 04:33:58.832263 3198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:58.851953 3198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:58.854844 3198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:58.855046 3198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:58.856460 3198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:58.856532 3198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:58.856601 3198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:58.856627 3198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:58 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:58.867686 3198 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:36875
I20260925 04:33:58.867718 3331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:36875 every 8 connection(s)
I20260925 04:33:58.868659 3198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:33:58.871714 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:58.857185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:52005 (local address 127.27.46.254:41867)
0925 04:33:58.857348 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:58.857355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:58.858306 (+ 951us) server_negotiation.cc:407] Connection header received
0925 04:33:58.859618 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:58.859624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:58.859697 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:58.859758 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:58.861636 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.862362 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.864201 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.864555 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.864640 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:58.868026 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:58.868030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:58.868033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:58.868065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:58.869680 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.870182 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.870186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.870188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.870271 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.870613 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.870616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.870618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.870811 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:58.870915 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:58.871404 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 04:33:58.871539 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:33:58.872474 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:58.857629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.193:52005)
0925 04:33:58.858127 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:58.858189 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:58.859343 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:58.860219 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:58.860230 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:58.860693 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:58.861437 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:58.861460 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.862475 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.862481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:58.864057 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:58.864072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.864683 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.864687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:58.864886 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:58.865568 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:58.865627 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:58.867832 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:58.869797 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:58.869807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:58.869821 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:58.870074 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:58.870362 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:58.870366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:58.870368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:58.870512 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:58.870915 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:58.870923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:58.871167 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 04:33:58.871346 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 04:33:58.871670 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260925 04:33:58.873716 3332 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41867
I20260925 04:33:58.873872 3332 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:58.874395 3332 heartbeater.cc:511] Master 127.27.46.254:41867 requested a full tablet report, sending...
I20260925 04:33:58.875941 3119 ts_manager.cc:194] Registered new tserver with Master: bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875)
I20260925 04:33:58.877063 3119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:52005
I20260925 04:33:58.877094 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3198
I20260925 04:33:58.877224 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:33:58.877633 27835 external_mini_cluster.cc:1428] Setting key 9972fe16ea11b76977b99be01c627f6d
I20260925 04:33:58.884377 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:58.875768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48727 (local address 127.27.46.254:41867)
0925 04:33:58.875872 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:58.875875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:58.877488 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 04:33:58.877653 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:58.877658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:58.877720 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:58.877806 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:58.878338 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.878896 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.879715 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.879935 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.880061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:58.881301 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:58.881304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:58.881306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:58.881326 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:58.882704 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.883193 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.883195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.883196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.883265 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:58.883625 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:58.883628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:58.883629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:58.883787 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:58.883861 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:58.884174 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 04:33:58.884266 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:33:58.884593 3338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:58.875704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.193:48727)
0925 04:33:58.877389 (+ 1685us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:58.877407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:58.877557 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:58.877902 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:58.877904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:58.878105 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:58.878224 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:58.878232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.879015 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.879018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:58.879599 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:58.879607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:58.880068 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:58.880071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:58.880160 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:58.880665 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:58.880683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:58.881159 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:58.882812 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:58.882819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:58.882824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:58.883098 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:58.883391 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:58.883397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:58.883401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:58.883523 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:58.883878 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:58.883881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:58.883997 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 04:33:58.884115 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:33:58.884280 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
I20260925 04:33:58.906713 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41867
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:59.016921 3341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:59.017174 3341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:59.017218 3341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:59.021090 3341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:59.021142 3341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:59.021179 3341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:59.021212 3341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:33:59.026353 3341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:59.027325 3341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:59.028299 3341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:59.037799 3347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:59.037799 3346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:59.037895 3349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:59.037984 3341 server_base.cc:1034] running on GCE node
I20260925 04:33:59.038419 3341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:59.038988 3341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:59.040158 3341 hybrid_clock.cc:648] HybridClock initialized: now 1790310839040120 us; error 41 us; skew 500 ppm
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:59.042893 3341 init.cc:377] Logged in from keytab as oryx/127.27.46.194@KRBTEST.COM (short username oryx)
I20260925 04:33:59.043745 3341 webserver.cc:460] Webserver started at http://127.27.46.194:43093/ using document root <none> and password file <none>
I20260925 04:33:59.044301 3341 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:59.044385 3341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:59.044607 3341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:59.046649 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "6b59fde6747842aa8b955ca97ccd88cd"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "f0dcb0f6233464f87f93d98a95863726"
server_key_iv: "d7c670ec0a66c7f5c2077576462c9ac1"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.047201 3341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "6b59fde6747842aa8b955ca97ccd88cd"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "f0dcb0f6233464f87f93d98a95863726"
server_key_iv: "d7c670ec0a66c7f5c2077576462c9ac1"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.050733 3341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:33:59.053026 3356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:59.054232 3341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:59.054342 3341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "6b59fde6747842aa8b955ca97ccd88cd"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "f0dcb0f6233464f87f93d98a95863726"
server_key_iv: "d7c670ec0a66c7f5c2077576462c9ac1"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.054433 3341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:59.074183 3341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:59.077059 3341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:59.077251 3341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:59.078699 3341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:59.078745 3341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:59.078783 3341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:59.078835 3341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:59.089826 3341 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37319
I20260925 04:33:59.089999 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37319 every 8 connection(s)
I20260925 04:33:59.090658 3341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:33:59.093271 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3341
I20260925 04:33:59.093393 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:33:59.093761 27835 external_mini_cluster.cc:1428] Setting key daf69adc091e4ed255b9f3a0bfac1d0c
I20260925 04:33:59.094111 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.079698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:38533 (local address 127.27.46.254:41867)
0925 04:33:59.079846 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.079853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.080884 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 04:33:59.082165 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.082169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.082237 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.082341 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.084087 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.084990 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.086865 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.087083 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.087170 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.090341 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.090345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.090348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.090384 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.091988 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.092568 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.092570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.092572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.092647 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.092967 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.092969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.092970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.093142 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.093212 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.093880 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.093983 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:33:59.095088 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.080251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.194:38533)
0925 04:33:59.080718 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.080760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.081906 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.082608 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.082621 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.083111 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:59.083884 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.083907 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.085138 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.085144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.086728 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.086739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.087239 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.087245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.087461 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.088217 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:59.088242 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:59.090150 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:59.092092 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.092096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.092106 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.092446 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.092729 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.092732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.092733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.092879 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.093245 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:59.093252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:59.093595 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.093806 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.094135 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":135,"threads_started":1}
I20260925 04:33:59.098264 3480 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41867
I20260925 04:33:59.098467 3480 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:59.099004 3480 heartbeater.cc:511] Master 127.27.46.254:41867 requested a full tablet report, sending...
I20260925 04:33:59.100116 3119 ts_manager.cc:194] Registered new tserver with Master: 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319)
I20260925 04:33:59.100708 3119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:38533
WARNING: no policy specified for oryx/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:33:59.107108 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.098518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.194:59593)
0925 04:33:59.098948 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.098964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.099125 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.099420 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.099423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.099651 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:59.099792 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.099801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.100957 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.100964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.102103 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.102115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.102717 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.102720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.102824 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.103319 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:59.103334 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:59.103785 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:59.105677 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.105680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.105682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.105861 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.106125 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.106127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.106128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.106198 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.106610 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:59.106612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:59.106729 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.106821 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.106964 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260925 04:33:59.107108 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.098589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:59593 (local address 127.27.46.254:41867)
0925 04:33:59.098723 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.098730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.099047 (+ 317us) server_negotiation.cc:407] Connection header received
0925 04:33:59.099242 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.099247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.099302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.099385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.099939 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.100790 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.102243 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.102573 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.102692 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.103943 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.103946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.103948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.103971 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.105548 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.105949 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.105951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.105952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.106035 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.106272 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.106274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.106276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.106494 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.106582 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.106897 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.106996 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
I20260925 04:33:59.124656 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41867
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:33:59.239802 3489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:33:59.240108 3489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:33:59.240168 3489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:33:59.244283 3489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:33:59.244354 3489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:33:59.244400 3489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:33:59.244434 3489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:33:59.249993 3489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.46.254:41867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:33:59.251224 3489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:33:59.252197 3489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:33:59.261886 3498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:59.261899 3496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:33:59.261902 3495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:33:59.262125 3489 server_base.cc:1034] running on GCE node
I20260925 04:33:59.262586 3489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:33:59.263203 3489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:33:59.264358 3489 hybrid_clock.cc:648] HybridClock initialized: now 1790310839264325 us; error 45 us; skew 500 ppm
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:33:59.267104 3489 init.cc:377] Logged in from keytab as oryx/127.27.46.195@KRBTEST.COM (short username oryx)
I20260925 04:33:59.267963 3489 webserver.cc:460] Webserver started at http://127.27.46.195:33899/ using document root <none> and password file <none>
I20260925 04:33:59.268512 3489 fs_manager.cc:359] Metadata directory not provided
I20260925 04:33:59.268596 3489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:33:59.268816 3489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:33:59.270902 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "568c742e754b4a9292a7b16cf87a6f19"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "29596342ba1c96ca12e5037adfc9661c"
server_key_iv: "ab9bdc6ceeb0cabbe502d26a4159a9c8"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.271487 3489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "568c742e754b4a9292a7b16cf87a6f19"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "29596342ba1c96ca12e5037adfc9661c"
server_key_iv: "ab9bdc6ceeb0cabbe502d26a4159a9c8"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.275643 3489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 04:33:59.278402 3506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:59.280079 3489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:33:59.280332 3489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "568c742e754b4a9292a7b16cf87a6f19"
format_stamp: "Formatted at 2026-09-25 04:33:59 on dist-test-slave-r8r3"
server_key: "29596342ba1c96ca12e5037adfc9661c"
server_key_iv: "ab9bdc6ceeb0cabbe502d26a4159a9c8"
server_key_version: "encryptionkey@0"
I20260925 04:33:59.280457 3489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:33:59.303851 3489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:33:59.307310 3489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:33:59.307526 3489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:33:59.308986 3489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:33:59.309059 3489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:33:59.309131 3489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:33:59.309172 3489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:59.321183 3489 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:33993
I20260925 04:33:59.321667 3626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:33993 every 8 connection(s)
I20260925 04:33:59.322353 3489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:33:59.324050 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.310570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.195:46393)
0925 04:33:59.311121 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.311165 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.312108 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.313032 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.313044 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.313432 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:59.314066 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.314080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.314876 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.314879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.315850 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.315859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.316373 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.316378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.316588 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.317167 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:59.317186 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:59.319150 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:59.321328 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.321333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.321347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.321700 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.322011 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.322014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.322016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.322153 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.322565 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:59.322571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:59.322855 (+ 284us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.323049 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.323323 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":140,"threads_started":1}
I20260925 04:33:59.324717 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.309989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:46393 (local address 127.27.46.254:41867)
0925 04:33:59.310114 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.310118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.311265 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 04:33:59.312617 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.312622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.312693 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.312799 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.314218 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.314751 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.315985 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.316207 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.316320 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.319327 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.319332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.319335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.319388 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.321211 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.321814 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.321822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.321824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.321912 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.322254 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.322257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.322259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.322473 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.322569 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.324477 (+ 1908us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.324602 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:33:59.325637 3627 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41867
I20260925 04:33:59.325791 3627 heartbeater.cc:464] Registering TS with master...
I20260925 04:33:59.326267 3627 heartbeater.cc:511] Master 127.27.46.254:41867 requested a full tablet report, sending...
I20260925 04:33:59.327278 3123 ts_manager.cc:194] Registered new tserver with Master: 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:33:59.327927 3123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:46393
I20260925 04:33:59.331471 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3489
I20260925 04:33:59.331607 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:33:59.331938 27835 external_mini_cluster.cc:1428] Setting key 037349689036bce038cf2950f5e34c36
I20260925 04:33:59.333992 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:33:59.334875 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.327798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.27.46.195:46503)
0925 04:33:59.328127 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.328138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.328259 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.328531 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.328534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.328743 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:59.328866 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.328873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.329620 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.329623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.330289 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.330297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.330714 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.330717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.330783 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.331228 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:59.331245 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:59.331680 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:59.333371 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.333374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.333376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.333640 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.333896 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.333900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.333902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.333994 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.334345 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:59.334348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:59.334443 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.334549 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.334686 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":6,"thread_start_us":105,"threads_started":1}
I20260925 04:33:59.334903 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.327873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:46503 (local address 127.27.46.254:41867)
0925 04:33:59.327997 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.328004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.328230 (+ 226us) server_negotiation.cc:407] Connection header received
0925 04:33:59.328362 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.328365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.328412 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.328495 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.329006 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.329473 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.330421 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.330622 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.330694 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.331808 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.331811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.331813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.331833 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.333249 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.333725 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.333728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.333730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.333806 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.334083 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.334088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.334090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.334253 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.334356 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.334603 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.334707 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:33:59.358950 3174 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:33:59.352221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.27.46.254:41867)
0925 04:33:59.352355 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.352359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.352372 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:33:59.352416 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.352418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.352460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.352544 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.353280 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.354267 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.355124 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.355349 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.355436 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.358794 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45430: BlockingRecv error: recv got EOF from 127.0.0.1:45430 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:59.368315 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.360555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.27.46.254:41867)
0925 04:33:59.360663 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.360666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.360811 (+ 145us) server_negotiation.cc:407] Connection header received
0925 04:33:59.360911 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.360913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.360952 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.361019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.361631 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.362226 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.362981 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.363199 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.363281 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.365439 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.365442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.365445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.365467 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.366961 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.367313 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.367316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.367317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.367390 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.367616 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.367619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.367620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.367842 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.367963 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.368109 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.368197 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 04:33:59.371363 3123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:33:59.375084 3123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:59.390404 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.384293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.27.46.193:36875)
0925 04:33:59.384743 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.384753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.384931 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:33:59.385016 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.385021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.385222 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.385367 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.385775 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.386982 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.388922 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.389641 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.389750 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.389899 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.390154 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.390256 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":102,"threads_started":1}
I20260925 04:33:59.390412 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.383867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.0.0.1:33980)
0925 04:33:59.384725 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.384739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.384837 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.385541 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.385553 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.385581 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.385653 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.385661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.387154 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.387160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.388767 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.388776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.389769 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.389774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.389908 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.389959 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.390088 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.390215 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":353,"spinlock_wait_cycles":2816,"thread_start_us":103,"threads_started":1}
I20260925 04:33:59.391002 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.383658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37319 (local address 127.0.0.1:47770)
0925 04:33:59.384116 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.384166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.384448 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.385390 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.385396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.385418 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.385614 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.385621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.387150 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.387156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.389078 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.389090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.390206 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.390210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.390300 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.390324 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.390373 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.390447 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":29,"thread_start_us":59,"threads_started":1}
I20260925 04:33:59.391343 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.384865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33993 (local address 127.0.0.1:50720)
0925 04:33:59.385416 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.385433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.385586 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.386410 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.386414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.386435 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.386602 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.386611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.388277 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.388281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.389440 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.389449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.390875 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.390879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.390994 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.391022 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.391103 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.391177 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"spinlock_wait_cycles":2432,"thread_start_us":110,"threads_started":1}
I20260925 04:33:59.391659 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.385002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.27.46.195:33993)
0925 04:33:59.385894 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.385910 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.385936 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:33:59.386088 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.386095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.386263 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.386431 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.386790 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.388074 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.389603 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.390745 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.390880 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.391070 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.391257 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.391384 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":121,"threads_started":1}
I20260925 04:33:59.392217 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.383833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.27.46.194:37319)
0925 04:33:59.384953 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.384966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.384985 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:33:59.385059 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.385063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.385222 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.385771 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.385905 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.386989 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.389221 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.390043 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.391685 (+ 1642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.391924 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.391973 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.392074 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":114,"threads_started":1}
I20260925 04:33:59.393627 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 (DEFAULT_TABLE table=test-table [id=f5d53738284048d88bad219415bc4ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.394804 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae905a694e470fbe7cb9b7dfb8b093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.394732 3559 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 (DEFAULT_TABLE table=test-table [id=f5d53738284048d88bad219415bc4ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.395081 3406 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 (DEFAULT_TABLE table=test-table [id=f5d53738284048d88bad219415bc4ac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.395777 3559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae905a694e470fbe7cb9b7dfb8b093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.396239 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae905a694e470fbe7cb9b7dfb8b093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.403206 3656 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Bootstrap starting.
I20260925 04:33:59.404017 3655 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Bootstrap starting.
I20260925 04:33:59.405864 3657 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Bootstrap starting.
I20260925 04:33:59.405948 3655 tablet_bootstrap.cc:654] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.405970 3656 tablet_bootstrap.cc:654] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.406831 3655 log.cc:824] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:59.407604 3656 log.cc:824] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:59.407931 3657 tablet_bootstrap.cc:654] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.408650 3657 log.cc:824] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Log is configured to *not* fsync() on all Append() calls
I20260925 04:33:59.409096 3655 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: No bootstrap required, opened a new log
I20260925 04:33:59.409313 3655 ts_tablet_manager.cc:1397] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:33:59.409677 3656 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: No bootstrap required, opened a new log
I20260925 04:33:59.409865 3656 ts_tablet_manager.cc:1397] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:33:59.410737 3657 tablet_bootstrap.cc:492] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: No bootstrap required, opened a new log
I20260925 04:33:59.410919 3657 ts_tablet_manager.cc:1397] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:33:59.413895 3656 raft_consensus.cc:348] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.413815 3655 raft_consensus.cc:348] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.414157 3656 raft_consensus.cc:374] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.414160 3655 raft_consensus.cc:374] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.414216 3656 raft_consensus.cc:729] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.414223 3655 raft_consensus.cc:729] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.414721 3655 consensus_queue.cc:260] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.414746 3656 consensus_queue.cc:260] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.414841 3657 raft_consensus.cc:348] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.415091 3657 raft_consensus.cc:374] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.415163 3657 raft_consensus.cc:729] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.415635 3332 heartbeater.cc:503] Master 127.27.46.254:41867 was elected leader, sending a full tablet report...
I20260925 04:33:59.415827 3656 ts_tablet_manager.cc:1428] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:33:59.415868 3655 ts_tablet_manager.cc:1428] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:33:59.416170 3627 heartbeater.cc:503] Master 127.27.46.254:41867 was elected leader, sending a full tablet report...
I20260925 04:33:59.416683 3657 consensus_queue.cc:260] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.417430 3480 heartbeater.cc:503] Master 127.27.46.254:41867 was elected leader, sending a full tablet report...
I20260925 04:33:59.417667 3657 ts_tablet_manager.cc:1428] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:33:59.532248 3662 raft_consensus.cc:482] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:59.532601 3662 raft_consensus.cc:504] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.533936 3662 leader_election.cc:288] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875)
I20260925 04:33:59.538669 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.534413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:56901 (local address 127.27.46.194:37319)
0925 04:33:59.534550 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.534555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.534574 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:33:59.534640 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.534643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.534696 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.534787 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.535137 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.536010 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.537531 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.538118 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.538243 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.538302 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.538503 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.538566 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:33:59.538694 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.534266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37319 (local address 127.27.46.195:56901)
0925 04:33:59.534406 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.534424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.534539 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.534800 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.534803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.534821 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.534959 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.534965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.536151 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.536154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.537359 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.537368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.538238 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.538241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.538359 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.538388 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.538474 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.538553 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:33:59.539520 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.534275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.27.46.195:57487)
0925 04:33:59.534741 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.534753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.534935 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.535253 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.535256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.535275 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.535818 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.535827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.537135 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.537137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.538256 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.538266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.539146 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.539150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.539255 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.539274 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.539337 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.539386 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":117,"thread_start_us":151,"threads_started":1}
I20260925 04:33:59.539561 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.534396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:57487 (local address 127.27.46.193:36875)
0925 04:33:59.534545 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.534553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.534856 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:33:59.535073 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.535078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.535142 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.535245 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.536151 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.537001 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.538389 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.539011 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.539105 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.539141 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.539370 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.539446 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:33:59.539544 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae905a694e470fbe7cb9b7dfb8b093" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd" is_pre_election: true
I20260925 04:33:59.539935 3429 raft_consensus.cc:2454] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 0.
I20260925 04:33:59.540163 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae905a694e470fbe7cb9b7dfb8b093" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" is_pre_election: true
I20260925 04:33:59.540479 3286 raft_consensus.cc:2454] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 0.
I20260925 04:33:59.540496 3510 leader_election.cc:302] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd; no voters:
I20260925 04:33:59.540783 3662 raft_consensus.cc:2781] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:59.540901 3662 raft_consensus.cc:482] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:59.540957 3662 raft_consensus.cc:3037] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.542024 3662 raft_consensus.cc:504] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.542348 3662 leader_election.cc:288] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 election: Requested vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875)
I20260925 04:33:59.542673 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae905a694e470fbe7cb9b7dfb8b093" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
I20260925 04:33:59.542672 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae905a694e470fbe7cb9b7dfb8b093" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd"
I20260925 04:33:59.542791 3286 raft_consensus.cc:3037] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.542791 3429 raft_consensus.cc:3037] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.544114 3286 raft_consensus.cc:2454] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 1.
I20260925 04:33:59.544114 3429 raft_consensus.cc:2454] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 1.
I20260925 04:33:59.544445 3510 leader_election.cc:302] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd; no voters:
I20260925 04:33:59.544664 3662 raft_consensus.cc:2781] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:59.544893 3662 raft_consensus.cc:686] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 LEADER]: Becoming Leader. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Running, Role: LEADER
I20260925 04:33:59.545261 3662 consensus_queue.cc:237] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.548491 3119 catalog_manager.cc:5495] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "568c742e754b4a9292a7b16cf87a6f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } health_report { overall_health: HEALTHY } } }
W20260925 04:33:59.573385 3628 tablet.cc:2369] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:59.581357 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.577837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.27.46.195:33993)
0925 04:33:59.577980 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.577984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.577998 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:33:59.578165 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.578168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.578229 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.578327 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:59.578587 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.579102 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.579929 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.580154 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.580239 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.580476 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.580924 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.581012 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.581109 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.581165 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:33:59.587481 3429 raft_consensus.cc:1261] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Refusing update from remote peer 568c742e754b4a9292a7b16cf87a6f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.587481 3286 raft_consensus.cc:1261] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Refusing update from remote peer 568c742e754b4a9292a7b16cf87a6f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.588227 3662 consensus_queue.cc:1035] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:59.588438 3665 consensus_queue.cc:1035] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:33:59.592123 3481 tablet.cc:2369] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:59.597558 3670 mvcc.cc:205] Tried to move back new op lower bound from 7333113198942617600 to 7333113198780387328. Current Snapshot: MvccSnapshot[applied={T|T < 7333113198942617600}]
I20260925 04:33:59.597738 3668 mvcc.cc:205] Tried to move back new op lower bound from 7333113198942617600 to 7333113198780387328. Current Snapshot: MvccSnapshot[applied={T|T < 7333113198942617600}]
I20260925 04:33:59.605929 3119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:59.606113 3119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:59.608609 3119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Sending DeleteTablet for 3 replicas of tablet 73ae905a694e470fbe7cb9b7dfb8b093
I20260925 04:33:59.609413 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:33980
I20260925 04:33:59.609522 3406 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47770
I20260925 04:33:59.609546 3559 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ae905a694e470fbe7cb9b7dfb8b093 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 04:33:59.609973 3680 tablet_replica.cc:330] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: stopping tablet replica
I20260925 04:33:59.610183 3679 tablet_replica.cc:330] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: stopping tablet replica
I20260925 04:33:59.610303 3680 raft_consensus.cc:2229] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:59.610517 3679 raft_consensus.cc:2229] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:59.610626 3680 raft_consensus.cc:2258] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:59.610733 3119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:33:59.611008 3679 raft_consensus.cc:2258] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:33:59.611181 3119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:59.612433 3679 ts_tablet_manager.cc:1905] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.612630 3680 ts_tablet_manager.cc:1905] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.615485 3680 ts_tablet_manager.cc:1918] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:59.615586 3680 log.cc:1196] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/73ae905a694e470fbe7cb9b7dfb8b093
I20260925 04:33:59.615847 3678 tablet_replica.cc:330] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: stopping tablet replica
I20260925 04:33:59.615912 3680 ts_tablet_manager.cc:1939] T 73ae905a694e470fbe7cb9b7dfb8b093 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting consensus metadata
I20260925 04:33:59.616318 3678 raft_consensus.cc:2229] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:59.616437 3679 ts_tablet_manager.cc:1918] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:59.616809 3406 tablet_service.cc:1467] Processing CreateTablet for tablet 732da9df1ebc4342924f389e11f266a0 (DEFAULT_TABLE table=test-table [id=939aaaac2d654feca8e2b7e530996e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.616809 3559 tablet_service.cc:1467] Processing CreateTablet for tablet 732da9df1ebc4342924f389e11f266a0 (DEFAULT_TABLE table=test-table [id=939aaaac2d654feca8e2b7e530996e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.617054 3559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732da9df1ebc4342924f389e11f266a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.617054 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732da9df1ebc4342924f389e11f266a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.617123 3109 catalog_manager.cc:4841] TS 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319): tablet 73ae905a694e470fbe7cb9b7dfb8b093 (table test-table [id=f5d53738284048d88bad219415bc4ac7]) successfully deleted
I20260925 04:33:59.616600 3679 log.cc:1196] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/73ae905a694e470fbe7cb9b7dfb8b093
I20260925 04:33:59.617946 3679 ts_tablet_manager.cc:1939] T 73ae905a694e470fbe7cb9b7dfb8b093 P 568c742e754b4a9292a7b16cf87a6f19: Deleting consensus metadata
I20260925 04:33:59.618291 3678 raft_consensus.cc:2258] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:59.619227 3656 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Bootstrap starting.
I20260925 04:33:59.619227 3657 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Bootstrap starting.
I20260925 04:33:59.620199 3657 tablet_bootstrap.cc:654] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.620199 3656 tablet_bootstrap.cc:654] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.621469 3656 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: No bootstrap required, opened a new log
I20260925 04:33:59.621505 3109 catalog_manager.cc:4841] TS 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993): tablet 73ae905a694e470fbe7cb9b7dfb8b093 (table test-table [id=f5d53738284048d88bad219415bc4ac7]) successfully deleted
I20260925 04:33:59.621599 3656 ts_tablet_manager.cc:1397] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:59.621821 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 732da9df1ebc4342924f389e11f266a0 (DEFAULT_TABLE table=test-table [id=939aaaac2d654feca8e2b7e530996e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:33:59.621969 3656 raft_consensus.cc:348] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.622051 3656 raft_consensus.cc:374] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.622071 3656 raft_consensus.cc:729] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.622069 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732da9df1ebc4342924f389e11f266a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.622200 3656 consensus_queue.cc:260] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.622464 3656 ts_tablet_manager.cc:1428] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:33:59.623034 3657 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: No bootstrap required, opened a new log
I20260925 04:33:59.623104 3657 ts_tablet_manager.cc:1397] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:33:59.623571 3657 raft_consensus.cc:348] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.623687 3657 raft_consensus.cc:374] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.623723 3657 raft_consensus.cc:729] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.623857 3657 consensus_queue.cc:260] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.624114 3657 ts_tablet_manager.cc:1428] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:59.624666 3655 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Bootstrap starting.
I20260925 04:33:59.625701 3655 tablet_bootstrap.cc:654] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.626999 3655 tablet_bootstrap.cc:492] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: No bootstrap required, opened a new log
I20260925 04:33:59.627087 3655 ts_tablet_manager.cc:1397] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:33:59.627496 3655 raft_consensus.cc:348] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.627606 3655 raft_consensus.cc:374] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.627637 3655 raft_consensus.cc:729] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.627698 3678 ts_tablet_manager.cc:1905] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.627758 3655 consensus_queue.cc:260] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.628007 3655 ts_tablet_manager.cc:1428] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:59.630891 3678 ts_tablet_manager.cc:1918] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:33:59.631001 3678 log.cc:1196] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/73ae905a694e470fbe7cb9b7dfb8b093
I20260925 04:33:59.631322 3678 ts_tablet_manager.cc:1939] T 73ae905a694e470fbe7cb9b7dfb8b093 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting consensus metadata
I20260925 04:33:59.632382 3106 catalog_manager.cc:4841] TS bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875): tablet 73ae905a694e470fbe7cb9b7dfb8b093 (table test-table [id=f5d53738284048d88bad219415bc4ac7]) successfully deleted
I20260925 04:33:59.674988 3661 raft_consensus.cc:482] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:59.675172 3661 raft_consensus.cc:504] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.676548 3661 leader_election.cc:288] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:33:59.681281 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.676884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:38589 (local address 127.27.46.194:37319)
0925 04:33:59.677049 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.677056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.677134 (+ 78us) server_negotiation.cc:407] Connection header received
0925 04:33:59.677360 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.677363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.677423 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.677480 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.678048 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.678664 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.680075 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.680694 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.680842 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.680905 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.681085 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.681160 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:33:59.681313 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.676765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37319 (local address 127.27.46.193:38589)
0925 04:33:59.677034 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.677052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.677221 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.677681 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.677685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.677706 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.677871 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.677880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.678774 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.678778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.679935 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.679945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.680823 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.680827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.680942 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.680966 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.681055 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.681146 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260925 04:33:59.681548 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.677551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:35429 (local address 127.27.46.195:33993)
0925 04:33:59.677690 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.677696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.677713 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:33:59.677861 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.677865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.677930 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.677983 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:33:59.678397 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.679151 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.680380 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.680811 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.680922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.680975 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.681296 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.681384 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:33:59.681831 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732da9df1ebc4342924f389e11f266a0" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd" is_pre_election: true
I20260925 04:33:59.681998 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.677138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33993 (local address 127.27.46.193:35429)
0925 04:33:59.677617 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.677636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.677734 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.678125 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.678128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.678145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:33:59.678259 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.678264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.679282 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.679286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.680266 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.680273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.681010 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.681015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.681129 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.681156 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.681758 (+ 602us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.681848 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":96,"threads_started":1}
I20260925 04:33:59.681984 3429 raft_consensus.cc:2454] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 0.
I20260925 04:33:59.682487 3218 leader_election.cc:302] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b59fde6747842aa8b955ca97ccd88cd, bc4d25e323b9451c9f3a9d2d6fbcb011; no voters:
I20260925 04:33:59.682605 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732da9df1ebc4342924f389e11f266a0" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19" is_pre_election: true
I20260925 04:33:59.682786 3661 raft_consensus.cc:2781] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:59.682838 3661 raft_consensus.cc:482] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:59.682865 3661 raft_consensus.cc:3037] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.682898 3579 raft_consensus.cc:2454] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 0.
I20260925 04:33:59.683914 3661 raft_consensus.cc:504] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.684243 3661 leader_election.cc:288] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 election: Requested vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:33:59.684587 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732da9df1ebc4342924f389e11f266a0" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd"
I20260925 04:33:59.684716 3429 raft_consensus.cc:3037] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.684690 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732da9df1ebc4342924f389e11f266a0" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19"
I20260925 04:33:59.684795 3579 raft_consensus.cc:3037] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.685976 3429 raft_consensus.cc:2454] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 1.
I20260925 04:33:59.685976 3579 raft_consensus.cc:2454] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 1.
I20260925 04:33:59.686257 3218 leader_election.cc:302] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b59fde6747842aa8b955ca97ccd88cd, bc4d25e323b9451c9f3a9d2d6fbcb011; no voters:
I20260925 04:33:59.686514 3661 raft_consensus.cc:2781] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:59.686731 3661 raft_consensus.cc:686] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 LEADER]: Becoming Leader. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Running, Role: LEADER
I20260925 04:33:59.687001 3661 consensus_queue.cc:237] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.689615 3119 catalog_manager.cc:5495] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193). New cstate: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.254@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:33:59.756946 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.747297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41867 (local address 127.0.0.1:45454)
0925 04:33:59.747628 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.747649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.747755 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.748065 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.748067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.748447 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:33:59.748556 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.748563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.749645 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.749647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.750191 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.750198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.750759 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.750762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.750827 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.751436 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:33:59.751448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:33:59.753084 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:33:59.755321 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.755331 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.755338 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.755622 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.755884 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:33:59.755888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:33:59.755891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:33:59.755970 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:33:59.756348 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:33:59.756356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:33:59.756470 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.756590 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.756747 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260925 04:33:59.756945 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.747412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.27.46.254:41867)
0925 04:33:59.747578 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.747585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.747730 (+ 145us) server_negotiation.cc:407] Connection header received
0925 04:33:59.747844 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.747848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.747907 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.748026 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:33:59.748690 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.749532 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.750315 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.750634 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.750757 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.753240 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:33:59.753245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:33:59.753248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:33:59.753276 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:33:59.755209 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.755718 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.755723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.755725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.755800 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:33:59.756068 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:33:59.756071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:33:59.756072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:33:59.756242 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:33:59.756347 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.756635 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.756803 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":60}
I20260925 04:33:59.761270 3119 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:45454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:33:59.762058 3119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:33:59.767529 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7f69d2c394fd5acde6f8dd8619b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c80df6a580ea42cba92985571b2b7f06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:59.767972 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7f69d2c394fd5acde6f8dd8619b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.767845 3559 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7f69d2c394fd5acde6f8dd8619b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c80df6a580ea42cba92985571b2b7f06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:59.768181 3559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7f69d2c394fd5acde6f8dd8619b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.768070 3406 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf7f69d2c394fd5acde6f8dd8619b78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c80df6a580ea42cba92985571b2b7f06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:33:59.768465 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf7f69d2c394fd5acde6f8dd8619b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:33:59.770946 3656 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: Bootstrap starting.
I20260925 04:33:59.771042 3655 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: Bootstrap starting.
I20260925 04:33:59.771832 3656 tablet_bootstrap.cc:654] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.772292 3655 tablet_bootstrap.cc:654] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.773088 3656 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: No bootstrap required, opened a new log
I20260925 04:33:59.773172 3656 ts_tablet_manager.cc:1397] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:33:59.773697 3656 raft_consensus.cc:348] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.773818 3656 raft_consensus.cc:374] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.773842 3656 raft_consensus.cc:729] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.773934 3656 consensus_queue.cc:260] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.774067 3655 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: No bootstrap required, opened a new log
I20260925 04:33:59.774156 3655 ts_tablet_manager.cc:1397] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:59.774202 3656 ts_tablet_manager.cc:1428] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:33:59.774610 3655 raft_consensus.cc:348] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.774818 3673 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.774973 3673 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.774722 3655 raft_consensus.cc:374] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.775175 3655 raft_consensus.cc:729] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.775339 3655 consensus_queue.cc:260] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.776288 3657 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: Bootstrap starting.
I20260925 04:33:59.776592 3661 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.776736 3661 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.777379 3657 tablet_bootstrap.cc:654] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: Neither blocks nor log segments found. Creating new log.
I20260925 04:33:59.777380 3655 ts_tablet_manager.cc:1428] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:33:59.778776 3657 tablet_bootstrap.cc:492] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: No bootstrap required, opened a new log
I20260925 04:33:59.778867 3657 ts_tablet_manager.cc:1397] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:33:59.779325 3657 raft_consensus.cc:348] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.779444 3657 raft_consensus.cc:374] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:33:59.779487 3657 raft_consensus.cc:729] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Initialized, Role: FOLLOWER
I20260925 04:33:59.779601 3657 consensus_queue.cc:260] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.779906 3663 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.780076 3663 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.779896 3657 ts_tablet_manager.cc:1428] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260925 04:33:59.870117 3333 tablet.cc:2369] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:33:59.873839 3661 raft_consensus.cc:482] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:33:59.874007 3661 raft_consensus.cc:504] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.874465 3661 leader_election.cc:288] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:33:59.874919 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf7f69d2c394fd5acde6f8dd8619b78" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd" is_pre_election: true
I20260925 04:33:59.875077 3429 raft_consensus.cc:2454] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 0.
I20260925 04:33:59.875080 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf7f69d2c394fd5acde6f8dd8619b78" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19" is_pre_election: true
I20260925 04:33:59.875259 3579 raft_consensus.cc:2454] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 0.
I20260925 04:33:59.875360 3218 leader_election.cc:302] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b59fde6747842aa8b955ca97ccd88cd, bc4d25e323b9451c9f3a9d2d6fbcb011; no voters:
I20260925 04:33:59.875535 3661 raft_consensus.cc:2781] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:33:59.875608 3661 raft_consensus.cc:482] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:33:59.875669 3661 raft_consensus.cc:3037] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.876718 3661 raft_consensus.cc:504] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.877113 3661 leader_election.cc:288] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 election: Requested vote from peers 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:33:59.877548 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf7f69d2c394fd5acde6f8dd8619b78" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19"
I20260925 04:33:59.877552 3429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf7f69d2c394fd5acde6f8dd8619b78" candidate_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd"
I20260925 04:33:59.877668 3579 raft_consensus.cc:3037] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.877708 3429 raft_consensus.cc:3037] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:33:59.878664 3429 raft_consensus.cc:2454] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 1.
I20260925 04:33:59.878782 3579 raft_consensus.cc:2454] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4d25e323b9451c9f3a9d2d6fbcb011 in term 1.
I20260925 04:33:59.879001 3218 leader_election.cc:302] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b59fde6747842aa8b955ca97ccd88cd, bc4d25e323b9451c9f3a9d2d6fbcb011; no voters:
I20260925 04:33:59.879173 3661 raft_consensus.cc:2781] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:33:59.879271 3661 raft_consensus.cc:686] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 LEADER]: Becoming Leader. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Running, Role: LEADER
I20260925 04:33:59.879469 3661 consensus_queue.cc:237] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:33:59.880259 3661 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: TxnStatusTablet state changed. Reason: New leader bc4d25e323b9451c9f3a9d2d6fbcb011. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.880378 3661 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:59.880789 3698 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:33:59.880932 3698 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:33:59.881392 3119 catalog_manager.cc:5495] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193). New cstate: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20260925 04:33:59.902977 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.899259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.0.0.1:33986)
0925 04:33:59.899508 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.899523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.899638 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.899957 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.899960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.899968 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:59.900087 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.900092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.900797 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.900801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.901413 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.901421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.901892 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.901896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.901984 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.902036 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.902687 (+ 651us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.902703 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.902772 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.902859 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 04:33:59.902977 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.899368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.27.46.193:36875)
0925 04:33:59.899512 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.899519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.899567 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:33:59.899779 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.899782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.899837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.899945 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:59.900190 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.900691 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.901554 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.901779 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.901900 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.902189 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.902594 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.902714 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.902781 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.902833 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:33:59.906107 3429 raft_consensus.cc:1261] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Refusing update from remote peer bc4d25e323b9451c9f3a9d2d6fbcb011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.906226 3579 raft_consensus.cc:1261] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Refusing update from remote peer bc4d25e323b9451c9f3a9d2d6fbcb011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.906663 3661 consensus_queue.cc:1035] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:59.906847 3686 consensus_queue.cc:1035] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:59.908604 3673 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: TxnStatusTablet state changed. Reason: New leader bc4d25e323b9451c9f3a9d2d6fbcb011. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.908756 3673 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.908974 3663 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: TxnStatusTablet state changed. Reason: New leader bc4d25e323b9451c9f3a9d2d6fbcb011. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.909083 3663 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.910537 3686 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.910650 3686 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:59.911270 3669 mvcc.cc:205] Tried to move back new op lower bound from 7333113200250077184 to 7333113200147980288. Current Snapshot: MvccSnapshot[applied={T|T < 7333113200250077184}]
I20260925 04:33:59.911494 3663 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.911589 3663 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 6b59fde6747842aa8b955ca97ccd88cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.915096 3661 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.915202 3661 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P bc4d25e323b9451c9f3a9d2d6fbcb011: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:33:59.916133 3673 tablet_replica.cc:439] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } } }
I20260925 04:33:59.916263 3673 tablet_replica.cc:442] T 4bf7f69d2c394fd5acde6f8dd8619b78 P 568c742e754b4a9292a7b16cf87a6f19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:33:59.920421 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.916874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.27.46.193:36875)
0925 04:33:59.917296 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.917300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.917309 (+ 9us) server_negotiation.cc:407] Connection header received
0925 04:33:59.917342 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.917344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.917381 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.917462 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:59.917767 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.918538 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.919387 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.919622 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.919724 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.919980 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.920059 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.920134 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.920237 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.920287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260925 04:33:59.931396 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.927240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.27.46.193:40543)
0925 04:33:59.927542 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:33:59.927565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:33:59.927649 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:33:59.927887 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:33:59.927889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:33:59.927894 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:33:59.927985 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.927989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.929002 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.929004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.930135 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:33:59.930145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.930732 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.930735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:33:59.930799 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.930838 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.931141 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.931145 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:33:59.931184 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 04:33:59.931239 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260925 04:33:59.931407 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:33:59.927316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:40543 (local address 127.27.46.193:36875)
0925 04:33:59.927446 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:33:59.927451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:33:59.927635 (+ 184us) server_negotiation.cc:407] Connection header received
0925 04:33:59.927731 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:33:59.927734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:33:59.927787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:33:59.927872 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:33:59.928079 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.928908 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.930250 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:33:59.930621 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:33:59.930737 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:33:59.930963 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:33:59.931043 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:33:59.931124 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:33:59.931211 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 04:33:59.931268 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:33:59.937953 3579 raft_consensus.cc:1261] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Refusing update from remote peer bc4d25e323b9451c9f3a9d2d6fbcb011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.938071 3429 raft_consensus.cc:1261] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Refusing update from remote peer bc4d25e323b9451c9f3a9d2d6fbcb011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:33:59.938429 3661 consensus_queue.cc:1035] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:59.938576 3686 consensus_queue.cc:1035] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:33:59.981532 3123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:59.981704 3123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:33:59.983795 3123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Sending DeleteTablet for 3 replicas of tablet 732da9df1ebc4342924f389e11f266a0
I20260925 04:33:59.984380 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 732da9df1ebc4342924f389e11f266a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:33980
I20260925 04:33:59.984562 3678 tablet_replica.cc:330] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: stopping tablet replica
I20260925 04:33:59.984596 3406 tablet_service.cc:1514] Processing DeleteTablet for tablet 732da9df1ebc4342924f389e11f266a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47770
I20260925 04:33:59.984680 3678 raft_consensus.cc:2229] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:33:59.984723 3559 tablet_service.cc:1514] Processing DeleteTablet for tablet 732da9df1ebc4342924f389e11f266a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:33:59 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 04:33:59.984786 3680 tablet_replica.cc:330] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: stopping tablet replica
I20260925 04:33:59.984895 3680 raft_consensus.cc:2229] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:59.984923 3679 tablet_replica.cc:330] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: stopping tablet replica
I20260925 04:33:59.985008 3680 raft_consensus.cc:2258] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:59.984988 3678 raft_consensus.cc:2258] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:59.985057 3679 raft_consensus.cc:2229] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:33:59.985168 3679 raft_consensus.cc:2258] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:33:59.985922 3678 ts_tablet_manager.cc:1905] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.986135 3679 ts_tablet_manager.cc:1905] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.988518 3678 ts_tablet_manager.cc:1918] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:59.988597 3678 log.cc:1196] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/732da9df1ebc4342924f389e11f266a0
I20260925 04:33:59.988829 3678 ts_tablet_manager.cc:1939] T 732da9df1ebc4342924f389e11f266a0 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting consensus metadata
I20260925 04:33:59.988922 3679 ts_tablet_manager.cc:1918] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:59.989002 3679 log.cc:1196] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/732da9df1ebc4342924f389e11f266a0
I20260925 04:33:59.989243 3679 ts_tablet_manager.cc:1939] T 732da9df1ebc4342924f389e11f266a0 P 568c742e754b4a9292a7b16cf87a6f19: Deleting consensus metadata
I20260925 04:33:59.989439 3680 ts_tablet_manager.cc:1905] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:33:59.990514 3109 catalog_manager.cc:4841] TS 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993): tablet 732da9df1ebc4342924f389e11f266a0 (table test-table [id=939aaaac2d654feca8e2b7e530996e90]) successfully deleted
I20260925 04:33:59.991897 3680 ts_tablet_manager.cc:1918] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:33:59.991971 3680 log.cc:1196] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/732da9df1ebc4342924f389e11f266a0
I20260925 04:33:59.992242 3680 ts_tablet_manager.cc:1939] T 732da9df1ebc4342924f389e11f266a0 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting consensus metadata
I20260925 04:33:59.992895 3109 catalog_manager.cc:4841] TS 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319): tablet 732da9df1ebc4342924f389e11f266a0 (table test-table [id=939aaaac2d654feca8e2b7e530996e90]) successfully deleted
I20260925 04:33:59.993695 3106 catalog_manager.cc:4841] TS bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875): tablet 732da9df1ebc4342924f389e11f266a0 (table test-table [id=939aaaac2d654feca8e2b7e530996e90]) successfully deleted
Sep 25 04:33:59 dist-test-slave-r8r3 krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:34:00.019388 3174 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:00.012403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.27.46.254:41867)
0925 04:34:00.012568 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.012573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.012707 (+ 134us) server_negotiation.cc:407] Connection header received
0925 04:34:00.012835 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.012838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.012926 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.013032 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:00.013876 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.014733 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.015579 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.015933 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.016062 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.019225 (+ 3163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45456: BlockingRecv error: recv got EOF from 127.0.0.1:45456 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM
I20260925 04:34:00.029376 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.020894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.27.46.254:41867)
0925 04:34:00.021033 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.021037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.021130 (+ 93us) server_negotiation.cc:407] Connection header received
0925 04:34:00.021269 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.021273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.021317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.021397 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:00.022012 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.022500 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.023314 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.023526 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.023641 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.026161 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:00.026164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:00.026197 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:00.026227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:00.027680 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:00.028153 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:00.028155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:00.028157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:00.028231 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:00.028532 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:00.028534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:00.028535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:00.028699 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:00.028796 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.029105 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.029197 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:34:00.032109 3123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:34:00.032614 3123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:34:00.037683 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 (DEFAULT_TABLE table=test-table [id=cd71046f70324da08dd38131198a85dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.037804 3406 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 (DEFAULT_TABLE table=test-table [id=cd71046f70324da08dd38131198a85dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.038033 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b4a429ef541feb2f4c0b6381b7902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.038100 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b4a429ef541feb2f4c0b6381b7902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.038082 3559 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 (DEFAULT_TABLE table=test-table [id=cd71046f70324da08dd38131198a85dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.038353 3559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7b4a429ef541feb2f4c0b6381b7902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.040647 3656 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Bootstrap starting.
I20260925 04:34:00.040709 3655 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Bootstrap starting.
I20260925 04:34:00.041594 3655 tablet_bootstrap.cc:654] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.041594 3656 tablet_bootstrap.cc:654] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.042268 3657 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Bootstrap starting.
I20260925 04:34:00.042893 3655 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: No bootstrap required, opened a new log
I20260925 04:34:00.042995 3655 ts_tablet_manager.cc:1397] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:34:00.043092 3657 tablet_bootstrap.cc:654] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.043485 3655 raft_consensus.cc:348] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.043638 3655 raft_consensus.cc:374] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.043684 3655 raft_consensus.cc:729] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.043817 3655 consensus_queue.cc:260] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.044106 3655 ts_tablet_manager.cc:1428] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:34:00.044485 3657 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: No bootstrap required, opened a new log
I20260925 04:34:00.044579 3657 ts_tablet_manager.cc:1397] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:34:00.045091 3657 raft_consensus.cc:348] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.045207 3657 raft_consensus.cc:374] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.045244 3657 raft_consensus.cc:729] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.045364 3657 consensus_queue.cc:260] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.045529 3656 tablet_bootstrap.cc:492] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: No bootstrap required, opened a new log
I20260925 04:34:00.045617 3656 ts_tablet_manager.cc:1397] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 04:34:00.045677 3657 ts_tablet_manager.cc:1428] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:34:00.046044 3656 raft_consensus.cc:348] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.046157 3656 raft_consensus.cc:374] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.046195 3656 raft_consensus.cc:729] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.046320 3656 consensus_queue.cc:260] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.046590 3656 ts_tablet_manager.cc:1428] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 04:34:00.287711 3673 raft_consensus.cc:482] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:34:00.287896 3673 raft_consensus.cc:504] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.288444 3673 leader_election.cc:288] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875), 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319)
I20260925 04:34:00.288944 3428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b4a429ef541feb2f4c0b6381b7902" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd" is_pre_election: true
I20260925 04:34:00.288940 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b4a429ef541feb2f4c0b6381b7902" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" is_pre_election: true
I20260925 04:34:00.289127 3428 raft_consensus.cc:2454] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 0.
I20260925 04:34:00.289132 3286 raft_consensus.cc:2454] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 0.
I20260925 04:34:00.289430 3510 leader_election.cc:302] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd; no voters:
I20260925 04:34:00.289644 3673 raft_consensus.cc:2781] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:34:00.289709 3673 raft_consensus.cc:482] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:34:00.289737 3673 raft_consensus.cc:3037] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.290747 3673 raft_consensus.cc:504] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.291090 3673 leader_election.cc:288] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 election: Requested vote from peers bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875), 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319)
I20260925 04:34:00.291460 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b4a429ef541feb2f4c0b6381b7902" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
I20260925 04:34:00.291460 3428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7b4a429ef541feb2f4c0b6381b7902" candidate_uuid: "568c742e754b4a9292a7b16cf87a6f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b59fde6747842aa8b955ca97ccd88cd"
I20260925 04:34:00.291607 3286 raft_consensus.cc:3037] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.291618 3428 raft_consensus.cc:3037] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.292528 3428 raft_consensus.cc:2454] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 1.
I20260925 04:34:00.292613 3286 raft_consensus.cc:2454] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 568c742e754b4a9292a7b16cf87a6f19 in term 1.
I20260925 04:34:00.292820 3510 leader_election.cc:302] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd; no voters:
I20260925 04:34:00.293025 3673 raft_consensus.cc:2781] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:00.293154 3673 raft_consensus.cc:686] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 LEADER]: Becoming Leader. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Running, Role: LEADER
I20260925 04:34:00.293324 3673 consensus_queue.cc:237] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.295070 3119 catalog_manager.cc:5495] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "568c742e754b4a9292a7b16cf87a6f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } health_report { overall_health: HEALTHY } } }
W20260925 04:34:00.328280 3622 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.27.46.195:33993
I20260925 04:34:00.346065 3730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.342595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.27.46.195:33993)
0925 04:34:00.342843 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.342848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.342863 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:34:00.342915 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.342918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.342972 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.343057 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:34:00.343304 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.344096 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.344972 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.345202 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.345291 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.345604 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.345679 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.345734 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.345814 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.345861 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260925 04:34:00.348161 3428 raft_consensus.cc:1261] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Refusing update from remote peer 568c742e754b4a9292a7b16cf87a6f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:00.348206 3286 raft_consensus.cc:1261] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Refusing update from remote peer 568c742e754b4a9292a7b16cf87a6f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:00.348615 3673 consensus_queue.cc:1035] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:00.348840 3727 consensus_queue.cc:1035] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:00.356233 3123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:00.356370 3123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:00.358381 3123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Sending DeleteTablet for 3 replicas of tablet 3b7b4a429ef541feb2f4c0b6381b7902
I20260925 04:34:00.358965 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:33980
I20260925 04:34:00.358984 3559 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 04:34:00.359086 3406 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7b4a429ef541feb2f4c0b6381b7902 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47770
I20260925 04:34:00.359207 3678 tablet_replica.cc:330] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: stopping tablet replica
I20260925 04:34:00.359337 3680 tablet_replica.cc:330] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: stopping tablet replica
I20260925 04:34:00.359361 3678 raft_consensus.cc:2229] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:34:00.359452 3680 raft_consensus.cc:2229] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:34:00.359488 3678 raft_consensus.cc:2258] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:00.359586 3680 raft_consensus.cc:2258] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:00.359651 3679 tablet_replica.cc:330] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: stopping tablet replica
I20260925 04:34:00.359777 3679 raft_consensus.cc:2229] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:34:00.359685 3123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:34:00.359964 3679 raft_consensus.cc:2258] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:34:00.360107 3123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:34:00.360276 3680 ts_tablet_manager.cc:1905] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.360839 3678 ts_tablet_manager.cc:1905] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.362800 3680 ts_tablet_manager.cc:1918] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:34:00.362895 3680 log.cc:1196] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/3b7b4a429ef541feb2f4c0b6381b7902
I20260925 04:34:00.363117 3678 ts_tablet_manager.cc:1918] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:34:00.363174 3680 ts_tablet_manager.cc:1939] T 3b7b4a429ef541feb2f4c0b6381b7902 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting consensus metadata
I20260925 04:34:00.363189 3678 log.cc:1196] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/3b7b4a429ef541feb2f4c0b6381b7902
I20260925 04:34:00.363445 3678 ts_tablet_manager.cc:1939] T 3b7b4a429ef541feb2f4c0b6381b7902 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting consensus metadata
I20260925 04:34:00.363933 3109 catalog_manager.cc:4841] TS 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319): tablet 3b7b4a429ef541feb2f4c0b6381b7902 (table test-table [id=cd71046f70324da08dd38131198a85dd]) successfully deleted
I20260925 04:34:00.364147 3106 catalog_manager.cc:4841] TS bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875): tablet 3b7b4a429ef541feb2f4c0b6381b7902 (table test-table [id=cd71046f70324da08dd38131198a85dd]) successfully deleted
I20260925 04:34:00.364423 3679 ts_tablet_manager.cc:1905] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.365249 3264 tablet_service.cc:1467] Processing CreateTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 (DEFAULT_TABLE table=test-table [id=5941154f107941729aa25d886c169d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.365249 3559 tablet_service.cc:1467] Processing CreateTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 (DEFAULT_TABLE table=test-table [id=5941154f107941729aa25d886c169d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.365546 3559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dcae91c04f4ab4add72f8fbb95b503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.365562 3264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dcae91c04f4ab4add72f8fbb95b503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.366744 3679 ts_tablet_manager.cc:1918] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:34:00.366827 3679 log.cc:1196] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/3b7b4a429ef541feb2f4c0b6381b7902
I20260925 04:34:00.367105 3679 ts_tablet_manager.cc:1939] T 3b7b4a429ef541feb2f4c0b6381b7902 P 568c742e754b4a9292a7b16cf87a6f19: Deleting consensus metadata
I20260925 04:34:00.367327 3655 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: Bootstrap starting.
I20260925 04:34:00.368033 3655 tablet_bootstrap.cc:654] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.368031 3656 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Bootstrap starting.
I20260925 04:34:00.368589 3406 tablet_service.cc:1467] Processing CreateTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 (DEFAULT_TABLE table=test-table [id=5941154f107941729aa25d886c169d95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:00.368862 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dcae91c04f4ab4add72f8fbb95b503. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:00.369045 3656 tablet_bootstrap.cc:654] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.369202 3109 catalog_manager.cc:4841] TS 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993): tablet 3b7b4a429ef541feb2f4c0b6381b7902 (table test-table [id=cd71046f70324da08dd38131198a85dd]) successfully deleted
I20260925 04:34:00.370193 3656 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: No bootstrap required, opened a new log
I20260925 04:34:00.370236 3655 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: No bootstrap required, opened a new log
I20260925 04:34:00.370297 3656 ts_tablet_manager.cc:1397] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:34:00.370318 3655 ts_tablet_manager.cc:1397] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 04:34:00.370654 3656 raft_consensus.cc:348] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.370769 3656 raft_consensus.cc:374] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.370723 3655 raft_consensus.cc:348] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.370810 3656 raft_consensus.cc:729] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568c742e754b4a9292a7b16cf87a6f19, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.370832 3655 raft_consensus.cc:374] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.370886 3655 raft_consensus.cc:729] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4d25e323b9451c9f3a9d2d6fbcb011, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.370947 3656 consensus_queue.cc:260] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.371026 3655 consensus_queue.cc:260] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.371236 3656 ts_tablet_manager.cc:1428] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:34:00.371291 3655 ts_tablet_manager.cc:1428] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:34:00.371766 3657 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Bootstrap starting.
I20260925 04:34:00.372622 3657 tablet_bootstrap.cc:654] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:00.373921 3657 tablet_bootstrap.cc:492] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: No bootstrap required, opened a new log
I20260925 04:34:00.374014 3657 ts_tablet_manager.cc:1397] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:34:00.374506 3657 raft_consensus.cc:348] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.374658 3657 raft_consensus.cc:374] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:00.374691 3657 raft_consensus.cc:729] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Initialized, Role: FOLLOWER
I20260925 04:34:00.374819 3657 consensus_queue.cc:260] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.375103 3657 ts_tablet_manager.cc:1428] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:34:00.525580 3663 raft_consensus.cc:482] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:34:00.525712 3663 raft_consensus.cc:504] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.526949 3663 leader_election.cc:288] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:34:00.531241 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.527348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:56083 (local address 127.27.46.193:36875)
0925 04:34:00.527611 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.527618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.527757 (+ 139us) server_negotiation.cc:407] Connection header received
0925 04:34:00.527896 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.527899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.527939 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.528038 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:00.528381 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.528881 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530137 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.530592 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.530731 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.531002 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.531089 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260925 04:34:00.531244 3730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.527296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:42211 (local address 127.27.46.195:33993)
0925 04:34:00.527426 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.527432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.527682 (+ 250us) server_negotiation.cc:407] Connection header received
0925 04:34:00.527825 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.527840 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.527907 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.527974 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:00.528337 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.528907 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530253 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.530707 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530872 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.530933 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.531055 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.531131 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:34:00.531266 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.527224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:33993 (local address 127.27.46.194:42211)
0925 04:34:00.527566 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:00.527581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:00.527699 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:00.528029 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:00.528032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:00.528049 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:00.528212 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.528219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.529062 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.529065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.530122 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.530133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.530839 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.530913 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.530931 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:34:00.531034 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:34:00.531100 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":36,"thread_start_us":91,"threads_started":1}
I20260925 04:34:00.531270 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.527243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.27.46.194:56083)
0925 04:34:00.527627 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:00.527661 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:00.527781 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:00.528071 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:00.528075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:00.528097 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:00.528237 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.528242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.529023 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.529026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.530011 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.530022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.530742 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.530749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.530859 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.530890 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:34:00.530979 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 04:34:00.531069 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":52,"thread_start_us":122,"threads_started":1}
I20260925 04:34:00.531721 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dcae91c04f4ab4add72f8fbb95b503" candidate_uuid: "6b59fde6747842aa8b955ca97ccd88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19" is_pre_election: true
I20260925 04:34:00.531912 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dcae91c04f4ab4add72f8fbb95b503" candidate_uuid: "6b59fde6747842aa8b955ca97ccd88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" is_pre_election: true
I20260925 04:34:00.531932 3579 raft_consensus.cc:2454] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b59fde6747842aa8b955ca97ccd88cd in term 0.
I20260925 04:34:00.532063 3286 raft_consensus.cc:2454] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b59fde6747842aa8b955ca97ccd88cd in term 0.
I20260925 04:34:00.532411 3360 leader_election.cc:302] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd, bc4d25e323b9451c9f3a9d2d6fbcb011; no voters:
I20260925 04:34:00.532723 3663 raft_consensus.cc:2781] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:34:00.532780 3663 raft_consensus.cc:482] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:34:00.532807 3663 raft_consensus.cc:3037] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.533790 3663 raft_consensus.cc:504] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.534140 3663 leader_election.cc:288] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [CANDIDATE]: Term 1 election: Requested vote from peers bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875), 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993)
I20260925 04:34:00.534459 3286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dcae91c04f4ab4add72f8fbb95b503" candidate_uuid: "6b59fde6747842aa8b955ca97ccd88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011"
I20260925 04:34:00.534590 3286 raft_consensus.cc:3037] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.534541 3579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dcae91c04f4ab4add72f8fbb95b503" candidate_uuid: "6b59fde6747842aa8b955ca97ccd88cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568c742e754b4a9292a7b16cf87a6f19"
I20260925 04:34:00.534689 3579 raft_consensus.cc:3037] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:00.535599 3286 raft_consensus.cc:2454] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b59fde6747842aa8b955ca97ccd88cd in term 1.
I20260925 04:34:00.535905 3579 raft_consensus.cc:2454] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b59fde6747842aa8b955ca97ccd88cd in term 1.
I20260925 04:34:00.536212 3360 leader_election.cc:302] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568c742e754b4a9292a7b16cf87a6f19, 6b59fde6747842aa8b955ca97ccd88cd; no voters:
I20260925 04:34:00.536388 3663 raft_consensus.cc:2781] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:00.536582 3663 raft_consensus.cc:686] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 LEADER]: Becoming Leader. State: Replica: 6b59fde6747842aa8b955ca97ccd88cd, State: Running, Role: LEADER
I20260925 04:34:00.536890 3663 consensus_queue.cc:237] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } }
I20260925 04:34:00.539413 3122 catalog_manager.cc:5495] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194). New cstate: current_term: 1 leader_uuid: "6b59fde6747842aa8b955ca97ccd88cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b59fde6747842aa8b955ca97ccd88cd" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20260925 04:34:00.574978 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.570645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.27.46.194:37319)
0925 04:34:00.570940 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.570948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.570962 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:34:00.571038 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.571040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.571098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.571215 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:34:00.571445 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.572009 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.573174 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.573450 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.573644 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.573963 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.574484 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.574584 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.574674 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.574746 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":140,"threads_started":1}
I20260925 04:34:00.586433 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.582910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:32935 (local address 127.27.46.193:36875)
0925 04:34:00.583077 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.583082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.583260 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:34:00.583441 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.583445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.583498 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.583578 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:34:00.583853 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.584407 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.585306 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.585559 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.585666 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.585992 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.586066 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.586143 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.586231 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.586285 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:34:00.586453 3746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.582811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.27.46.194:32935)
0925 04:34:00.583162 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:00.583174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:00.583318 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:00.583599 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:00.583602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:00.583610 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:34:00.583747 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.583753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.584518 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.584523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.585185 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.585193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.585684 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.585687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.585791 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.585836 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.586147 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.586156 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:34:00.586221 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:34:00.586301 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260925 04:34:00.592713 3286 raft_consensus.cc:1261] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Refusing update from remote peer 6b59fde6747842aa8b955ca97ccd88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:00.592779 3579 raft_consensus.cc:1261] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Refusing update from remote peer 6b59fde6747842aa8b955ca97ccd88cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:00.593312 3663 consensus_queue.cc:1035] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "568c742e754b4a9292a7b16cf87a6f19" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:00.593556 3739 consensus_queue.cc:1035] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4d25e323b9451c9f3a9d2d6fbcb011" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:00.611901 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.608607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:35937 (local address 127.27.46.194:37319)
0925 04:34:00.608712 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:00.608717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:00.608737 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:34:00.608786 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:00.608789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:00.608858 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:00.608947 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:34:00.609177 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.609740 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.610803 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.611105 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.611210 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.611500 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.611589 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.611647 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:00.611737 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 04:34:00.611785 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:34:00.611929 3751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.608279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37319 (local address 127.27.46.193:35937)
0925 04:34:00.608561 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:00.608577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:00.608673 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:00.608968 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:00.608971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:00.608980 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:34:00.609094 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.609100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.609826 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.609829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.610687 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:00.610701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:00.611218 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:00.611222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:00.611318 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:00.611360 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:34:00.611667 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:34:00.611673 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:34:00.611723 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 04:34:00.611804 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260925 04:34:00.634718 3122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:00.634899 3122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:00.637229 3122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e5251a5a6d4341a776fe542f948133: Sending DeleteTablet for 3 replicas of tablet 97dcae91c04f4ab4add72f8fbb95b503
I20260925 04:34:00.637888 3559 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 04:34:00.638015 3406 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47770
I20260925 04:34:00.638082 3679 tablet_replica.cc:330] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: stopping tablet replica
I20260925 04:34:00.638201 3680 tablet_replica.cc:330] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: stopping tablet replica
I20260925 04:34:00.638197 3679 raft_consensus.cc:2229] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:34:00.638319 3680 raft_consensus.cc:2229] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:34:00.638330 3679 raft_consensus.cc:2258] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:00.638620 3680 raft_consensus.cc:2258] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:00.639144 3679 ts_tablet_manager.cc:1905] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.639192 3264 tablet_service.cc:1514] Processing DeleteTablet for tablet 97dcae91c04f4ab4add72f8fbb95b503 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:00 UTC) from {username='oryx', principal='oryx/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:33980
I20260925 04:34:00.639266 3680 ts_tablet_manager.cc:1905] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.639379 3678 tablet_replica.cc:330] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: stopping tablet replica
I20260925 04:34:00.639494 3678 raft_consensus.cc:2229] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:34:00.639599 3678 raft_consensus.cc:2258] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:00.640609 3678 ts_tablet_manager.cc:1905] T 97dcae91c04f4ab4add72f8fbb95b503 P bc4d25e323b9451c9f3a9d2d6fbcb011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:00.641674 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3198
I20260925 04:34:00.641870 3680 ts_tablet_manager.cc:1918] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:34:00.641929 3680 log.cc:1196] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/97dcae91c04f4ab4add72f8fbb95b503
I20260925 04:34:00.642134 3680 ts_tablet_manager.cc:1939] T 97dcae91c04f4ab4add72f8fbb95b503 P 6b59fde6747842aa8b955ca97ccd88cd: Deleting consensus metadata
I20260925 04:34:00.642977 3109 catalog_manager.cc:4841] TS 6b59fde6747842aa8b955ca97ccd88cd (127.27.46.194:37319): tablet 97dcae91c04f4ab4add72f8fbb95b503 (table test-table [id=5941154f107941729aa25d886c169d95]) successfully deleted
I20260925 04:34:00.643500 3679 ts_tablet_manager.cc:1918] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:34:00.643615 3679 log.cc:1196] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/97dcae91c04f4ab4add72f8fbb95b503
I20260925 04:34:00.643879 3679 ts_tablet_manager.cc:1939] T 97dcae91c04f4ab4add72f8fbb95b503 P 568c742e754b4a9292a7b16cf87a6f19: Deleting consensus metadata
I20260925 04:34:00.644532 3109 catalog_manager.cc:4841] TS 568c742e754b4a9292a7b16cf87a6f19 (127.27.46.195:33993): tablet 97dcae91c04f4ab4add72f8fbb95b503 (table test-table [id=5941154f107941729aa25d886c169d95]) successfully deleted
W20260925 04:34:00.652278 3106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:36875 (error 108)
I20260925 04:34:00.652526 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3341
I20260925 04:34:00.654627 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:00.654064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:36875 (local address 127.0.0.1:33998)
0925 04:34:00.654431 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:00.654495 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:36875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":68,"threads_started":1}
W20260925 04:34:00.654881 3106 catalog_manager.cc:4584] TS bc4d25e323b9451c9f3a9d2d6fbcb011 (127.27.46.193:36875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97dcae91c04f4ab4add72f8fbb95b503: Network error: Client connection negotiation failed: client connection to 127.27.46.193:36875: connect: Connection refused (error 111)
I20260925 04:34:00.660137 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3489
I20260925 04:34:00.666978 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3088
2026-09-25T04:34:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3494 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3759](info): set up 2 sockets
Sep 25 04:34:00 dist-test-slave-r8r3 krb5kdc[3759](info): commencing operation
krb5kdc: starting...
W20260925 04:34:01.754885 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:34:01 dist-test-slave-r8r3 krb5kdc[3759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:34:01.889465 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33827
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:42985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:02.007982 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:02.008363 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:02.008451 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:02.012728 3773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:02.012858 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:02.012897 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:02.012936 3773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:02.012969 3773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:02.018864 3773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:42985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:33827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:02.020131 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:02.021256 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:34:02.031214 3773 server_base.cc:1034] running on GCE node
W20260925 04:34:02.031157 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:02.031152 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:02.031152 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:02.032156 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:02.033430 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:02.034603 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790310842034588 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T04:34:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1380 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:34:02 dist-test-slave-r8r3 krb5kdc[3786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:34:02 dist-test-slave-r8r3 krb5kdc[3786](info): set up 2 sockets
Sep 25 04:34:02 dist-test-slave-r8r3 krb5kdc[3786](info): commencing operation
krb5kdc: starting...
W20260925 04:34:03.130524 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:34:03.260651 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38719
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:03.376924 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:03.377202 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:03.377244 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:03.381227 3799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:03.381286 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:03.381306 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:03.381323 3799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:03.381350 3799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:03.386554 3799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:03.387710 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:03.388753 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:03.397197 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:03.397223 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:03.397277 3799 server_base.cc:1034] running on GCE node
W20260925 04:34:03.397222 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:03.398098 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:03.399277 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:03.400455 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790310843400447 us; error 27 us; skew 500 ppm
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:03.403182 3799 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:34:03.404072 3799 webserver.cc:460] Webserver started at http://127.27.46.254:43723/ using document root <none> and password file <none>
I20260925 04:34:03.404627 3799 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:03.404686 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:03.404867 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:03.406976 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "aab286e68497465588efb5867d9c8536"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "952900a55f5ff46652eb4b1b7ecbe948"
server_key_iv: "e9369db9bb80b97277be3021da71693e"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.407505 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "aab286e68497465588efb5867d9c8536"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "952900a55f5ff46652eb4b1b7ecbe948"
server_key_iv: "e9369db9bb80b97277be3021da71693e"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.411235 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:34:03.413636 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.414844 3799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:03.414974 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "aab286e68497465588efb5867d9c8536"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "952900a55f5ff46652eb4b1b7ecbe948"
server_key_iv: "e9369db9bb80b97277be3021da71693e"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.415086 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:03.442553 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:03.445566 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:03.445775 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:03.453598 3799 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38719
I20260925 04:34:03.453611 3866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38719 every 8 connection(s)
I20260925 04:34:03.454602 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:34:03.457331 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:03.457782 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3799
I20260925 04:34:03.457914 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:34:03.458221 27835 external_mini_cluster.cc:1428] Setting key bf032a8f7575de4c78c1613154e1c362
I20260925 04:34:03.464564 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Bootstrap starting.
I20260925 04:34:03.467191 3867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Neither blocks nor log segments found. Creating new log.
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:03.468026 3867 log.cc:824] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:03.470134 3867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: No bootstrap required, opened a new log
I20260925 04:34:03.472905 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:03.459663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.27.46.254:38719)
0925 04:34:03.460078 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:03.460096 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:03.460147 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:34:03.460902 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:03.460926 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:03.461357 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:03.461749 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:03.462463 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.463986 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.464842 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.465488 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.465706 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:03.468111 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:03.468119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:03.468135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:03.468169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:03.470037 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.470513 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.470518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.470522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.470646 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.470942 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.470944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.470945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.471324 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:03.471532 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:03.471851 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 04:34:03.472138 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260925 04:34:03.474093 3867 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:03.474339 3867 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:03.474442 3867 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab286e68497465588efb5867d9c8536, State: Initialized, Role: FOLLOWER
I20260925 04:34:03.474882 3867 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:03.475032 3867 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:34:03.475117 3867 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:34:03.475239 3867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:03.476198 3867 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:03.476545 3867 leader_election.cc:302] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab286e68497465588efb5867d9c8536; no voters:
I20260925 04:34:03.476903 3867 leader_election.cc:288] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:34:03.477013 3872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:03.477276 3872 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 LEADER]: Becoming Leader. State: Replica: aab286e68497465588efb5867d9c8536, State: Running, Role: LEADER
I20260925 04:34:03.477659 3872 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:03.478225 3867 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:34:03.479350 3874 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab286e68497465588efb5867d9c8536. Latest consensus state: current_term: 1 leader_uuid: "aab286e68497465588efb5867d9c8536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } } }
I20260925 04:34:03.479480 3874 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:03.479763 3873 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab286e68497465588efb5867d9c8536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } } }
I20260925 04:34:03.479847 3873 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:03.479874 3881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:34:03.483037 3881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:34:03.489053 3881 catalog_manager.cc:1347] Generated new cluster ID: c23f755036a34e8aba1ed1cebdb8dd7e
I20260925 04:34:03.489147 3881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:34:03.501883 3881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:34:03.502830 3881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:34:03.508925 3881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Generated new TSK 0
I20260925 04:34:03.509685 3881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:34:03.545226 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:03.667245 3893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:03.667510 3893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:03.667555 3893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:03.671561 3893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:03.671622 3893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:03.671679 3893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:34:03.676954 3893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:03.678061 3893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:03.679163 3893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:03.689280 3899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:03.689280 3898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:03.689332 3901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:03.689337 3893 server_base.cc:1034] running on GCE node
I20260925 04:34:03.690018 3893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:03.690758 3893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:03.691941 3893 hybrid_clock.cc:648] HybridClock initialized: now 1790310843691909 us; error 49 us; skew 500 ppm
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:03.694729 3893 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:34:03.695633 3893 webserver.cc:460] Webserver started at http://127.27.46.193:44163/ using document root <none> and password file <none>
I20260925 04:34:03.696216 3893 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:03.696305 3893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:03.696534 3893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:03.698729 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "a6dfcf7c78694a559e0f7593568e3824"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "35d2f817a574031f01139c5f34c95752"
server_key_iv: "7a4ee5f3abd37b8726fec983161dfa73"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.699304 3893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "a6dfcf7c78694a559e0f7593568e3824"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "35d2f817a574031f01139c5f34c95752"
server_key_iv: "7a4ee5f3abd37b8726fec983161dfa73"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.703082 3893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:34:03.705443 3908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.706687 3893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:03.706811 3893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "a6dfcf7c78694a559e0f7593568e3824"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "35d2f817a574031f01139c5f34c95752"
server_key_iv: "7a4ee5f3abd37b8726fec983161dfa73"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.706918 3893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:03.739534 3893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:03.742533 3893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:03.742751 3893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:03.743357 3893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:03.744210 3893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:03.744280 3893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.744349 3893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:03.744392 3893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.754045 3893 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44621
I20260925 04:34:03.754081 4023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44621 every 8 connection(s)
I20260925 04:34:03.754943 3893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:34:03.762811 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 3893
I20260925 04:34:03.762987 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:34:03.763383 27835 external_mini_cluster.cc:1428] Setting key 1ff8d23d8f5e29352b39b6751ee37d78
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:03.768914 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:03.756767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:47383 (local address 127.27.46.254:38719)
0925 04:34:03.756908 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:03.756911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:03.757787 (+ 876us) server_negotiation.cc:407] Connection header received
0925 04:34:03.759109 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:03.759112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:03.759167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:03.759268 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:03.760430 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.760969 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.762181 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.762409 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.762519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:03.765279 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:03.765282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:03.765285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:03.765310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:03.766804 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.767312 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.767316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.767318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.767401 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.767740 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.767744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.767746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.767931 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:03.768038 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:03.768664 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 04:34:03.768781 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:34:03.770171 4026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:03.757150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:47383)
0925 04:34:03.757650 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:03.757692 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:03.758896 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:03.759437 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:03.759445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:03.759790 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:03.760288 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:03.760302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.761104 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.761107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:03.762032 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:03.762041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.762560 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.762565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:03.762733 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:03.763393 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:03.763427 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:03.765128 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:03.766920 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:03.766925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:03.766940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:03.767209 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:03.767495 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:03.767499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:03.767501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:03.767645 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:03.768037 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:03.768045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:03.768374 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 04:34:03.768603 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 04:34:03.768956 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260925 04:34:03.771478 4024 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38719
I20260925 04:34:03.771732 4024 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:03.772240 4024 heartbeater.cc:511] Master 127.27.46.254:38719 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:34:03.773895 3831 ts_manager.cc:194] Registered new tserver with Master: a6dfcf7c78694a559e0f7593568e3824 (127.27.46.193:44621)
I20260925 04:34:03.775125 3831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:47383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:34:03.792613 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:03.903474 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:03.903811 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:03.903887 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:03.907956 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:03.908069 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:03.908160 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:34:03.914064 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:03.915231 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:03.916312 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:03.926517 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:03.926599 4029 server_base.cc:1034] running on GCE node
W20260925 04:34:03.926543 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:03.926543 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:03.927143 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:03.927793 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:03.928972 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790310843928950 us; error 38 us; skew 500 ppm
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:03.931641 4029 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:34:03.932490 4029 webserver.cc:460] Webserver started at http://127.27.46.194:39597/ using document root <none> and password file <none>
I20260925 04:34:03.933015 4029 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:03.933071 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:03.933235 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:03.935316 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "c5339008b5d54d6589dd6610f2b7eac8"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "d8dda7d493313705da98f0bffbeb71d9"
server_key_iv: "6b4f1225b03d11617e0d075557a00e5c"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.935896 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "c5339008b5d54d6589dd6610f2b7eac8"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "d8dda7d493313705da98f0bffbeb71d9"
server_key_iv: "6b4f1225b03d11617e0d075557a00e5c"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.939782 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:34:03.942400 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.944185 4029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:34:03.944340 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c5339008b5d54d6589dd6610f2b7eac8"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "d8dda7d493313705da98f0bffbeb71d9"
server_key_iv: "6b4f1225b03d11617e0d075557a00e5c"
server_key_version: "encryptionkey@0"
I20260925 04:34:03.944479 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:03.963102 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:03.966728 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:03.966961 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:03.967626 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:03.968547 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:03.968633 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.968726 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:03.968776 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:03.981469 4029 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37745
I20260925 04:34:03.981552 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37745 every 8 connection(s)
I20260925 04:34:03.982497 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:34:03.989794 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4029
I20260925 04:34:03.989988 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:34:03.990309 27835 external_mini_cluster.cc:1428] Setting key f2f78dfeb91b1d2ff0b2da95d1c15bf3
Sep 25 04:34:03 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:03.997577 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:03.984401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:38243 (local address 127.27.46.254:38719)
0925 04:34:03.984535 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:03.984539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:03.985388 (+ 849us) server_negotiation.cc:407] Connection header received
0925 04:34:03.986783 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:03.986786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:03.986841 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:03.986940 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:03.988124 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.988691 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.989983 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.990266 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.990372 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:03.993111 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:03.993114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:03.993117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:03.993143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:03.994820 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.995557 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.995565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.995567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.995640 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:03.996073 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:03.996080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:03.996084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:03.996338 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:03.996443 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:03.997216 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 04:34:03.997360 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:34:03.998811 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:03.984797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:38243)
0925 04:34:03.985244 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:03.985287 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:03.986576 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:03.987116 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:03.987123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:03.987466 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:03.987985 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:03.988001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.988820 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.988823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:03.989813 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:03.989823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:03.990427 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:03.990431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:03.990601 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:03.991373 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:03.991394 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:03.992952 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:03.994966 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:03.994976 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:03.994996 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:03.995438 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:03.995740 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:03.995747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:03.995752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:03.995960 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:03.996465 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:03.996473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:03.996852 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 04:34:03.997139 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 04:34:03.997580 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260925 04:34:04.000204 4161 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38719
I20260925 04:34:04.000469 4161 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:04.001013 4161 heartbeater.cc:511] Master 127.27.46.254:38719 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:34:04.002218 3831 ts_manager.cc:194] Registered new tserver with Master: c5339008b5d54d6589dd6610f2b7eac8 (127.27.46.194:37745)
I20260925 04:34:04.002878 3831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:38243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:34:04.020612 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:04.134136 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:04.134402 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:04.134488 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:04.138558 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:04.138633 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:04.138715 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:34:04.144098 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:04.145180 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:04.146234 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:04.156256 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.156334 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.156312 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:04.156431 4166 server_base.cc:1034] running on GCE node
I20260925 04:34:04.156957 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:04.157632 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:04.158788 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790310844158771 us; error 26 us; skew 500 ppm
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310844, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:04.161370 4166 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:34:04.162274 4166 webserver.cc:460] Webserver started at http://127.27.46.195:37535/ using document root <none> and password file <none>
I20260925 04:34:04.162847 4166 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:04.162937 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:04.163172 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:04.165284 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "2149ed04cd58453a9ceda840dd9c7680"
format_stamp: "Formatted at 2026-09-25 04:34:04 on dist-test-slave-r8r3"
server_key: "2178c493383e5a13883303b55919e166"
server_key_iv: "c0389e8169372c884eb8b252b1afb026"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.165936 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "2149ed04cd58453a9ceda840dd9c7680"
format_stamp: "Formatted at 2026-09-25 04:34:04 on dist-test-slave-r8r3"
server_key: "2178c493383e5a13883303b55919e166"
server_key_iv: "c0389e8169372c884eb8b252b1afb026"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.169703 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:34:04.172117 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.173292 4166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:34:04.173439 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "2149ed04cd58453a9ceda840dd9c7680"
format_stamp: "Formatted at 2026-09-25 04:34:04 on dist-test-slave-r8r3"
server_key: "2178c493383e5a13883303b55919e166"
server_key_iv: "c0389e8169372c884eb8b252b1afb026"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.173615 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:04.186334 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:04.189670 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:04.189893 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:04.190757 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:04.192018 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:04.192088 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.192155 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:04.192195 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.203094 4166 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:38229
I20260925 04:34:04.203114 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:38229 every 8 connection(s)
I20260925 04:34:04.204038 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:34:04.207671 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4166
I20260925 04:34:04.207861 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:34:04.208235 27835 external_mini_cluster.cc:1428] Setting key 0b52eeb912147039a219299f7333cb4c
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310844, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:04.220371 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.205999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:35259 (local address 127.27.46.254:38719)
0925 04:34:04.206153 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.206159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.207144 (+ 985us) server_negotiation.cc:407] Connection header received
0925 04:34:04.208943 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.208949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.209023 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.209117 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.210854 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.211922 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.213080 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.213566 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.213702 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.216661 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:04.216664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:04.216666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:04.216691 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:04.218299 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:04.218989 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:04.218993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:04.218995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:04.219094 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:04.219399 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:04.219401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:04.219402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:04.219639 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:04.219718 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:04.220155 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 04:34:04.220256 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:34:04.221197 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.206543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:35259)
0925 04:34:04.206995 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.207039 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.208693 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.209442 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.209452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.210075 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.210711 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.210727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.212051 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.212054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.212950 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.212960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.213721 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.213724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.213875 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.214527 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.214543 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.216465 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:04.218408 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:04.218414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:04.218456 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:04.218832 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:04.219183 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:04.219186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:04.219187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:04.219318 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:04.219725 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:04.219729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:04.219953 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 04:34:04.220111 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:34:04.220355 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260925 04:34:04.222503 4296 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38719
I20260925 04:34:04.222757 4296 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:04.223264 4296 heartbeater.cc:511] Master 127.27.46.254:38719 requested a full tablet report, sending...
I20260925 04:34:04.224248 3830 ts_manager.cc:194] Registered new tserver with Master: 2149ed04cd58453a9ceda840dd9c7680 (127.27.46.195:38229)
I20260925 04:34:04.224772 3830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:35259
I20260925 04:34:04.233955 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
Entry for principal oryx/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab.
WARNING: no policy specified for oryx/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
Entry for principal oryx/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab.
WARNING: no policy specified for oryx/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
Entry for principal oryx/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab.
I20260925 04:34:04.310307 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3893
I20260925 04:34:04.317108 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4029
I20260925 04:34:04.323045 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4166
I20260925 04:34:04.330034 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 3799
I20260925 04:34:04.336704 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38719
--webserver_interface=127.27.46.254
--webserver_port=43723
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:04.447060 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:04.447381 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:04.447455 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:04.451443 4305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:04.451546 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:04.451594 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:04.451653 4305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:04.451694 4305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:04.456990 4305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=43723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:04.458073 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:04.459136 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:04.467820 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.467808 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:04.467871 4305 server_base.cc:1034] running on GCE node
W20260925 04:34:04.467830 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:04.468627 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:04.469879 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:04.471055 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790310844471040 us; error 34 us; skew 500 ppm
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310844, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:04.473819 4305 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:34:04.474701 4305 webserver.cc:460] Webserver started at http://127.27.46.254:43723/ using document root <none> and password file <none>
I20260925 04:34:04.475260 4305 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:04.475364 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:04.479149 4305 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 04:34:04.481181 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.482522 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:34:04.482661 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "aab286e68497465588efb5867d9c8536"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "952900a55f5ff46652eb4b1b7ecbe948"
server_key_iv: "e9369db9bb80b97277be3021da71693e"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.483191 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:04.512602 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:04.515604 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:04.515810 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:04.522872 4305 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38719
I20260925 04:34:04.522902 4372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38719 every 8 connection(s)
I20260925 04:34:04.524026 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:34:04.528237 4373 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:34:04.529969 4373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Bootstrap starting.
I20260925 04:34:04.533481 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4305
I20260925 04:34:04.533967 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:44621
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:38719
--webserver_port=44163
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:34:04.550935 4373 log.cc:824] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:04.559307 4373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:34:04.559726 4373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Bootstrap complete.
I20260925 04:34:04.565388 4373 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:04.565928 4373 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab286e68497465588efb5867d9c8536, State: Initialized, Role: FOLLOWER
I20260925 04:34:04.566478 4373 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:04.566635 4373 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:34:04.566735 4373 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:34:04.566828 4373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:34:04.569360 4373 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:04.569758 4373 leader_election.cc:302] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab286e68497465588efb5867d9c8536; no voters:
I20260925 04:34:04.570113 4373 leader_election.cc:288] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:34:04.570255 4377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:34:04.570544 4377 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [term 2 LEADER]: Becoming Leader. State: Replica: aab286e68497465588efb5867d9c8536, State: Running, Role: LEADER
I20260925 04:34:04.570858 4377 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } }
I20260925 04:34:04.571367 4373 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:34:04.572376 4379 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab286e68497465588efb5867d9c8536. Latest consensus state: current_term: 2 leader_uuid: "aab286e68497465588efb5867d9c8536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } } }
I20260925 04:34:04.572396 4378 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aab286e68497465588efb5867d9c8536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab286e68497465588efb5867d9c8536" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38719 } } }
I20260925 04:34:04.572518 4379 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:04.572518 4378 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:04.576572 4392 catalog_manager.cc:1259] Loaded cluster ID: c23f755036a34e8aba1ed1cebdb8dd7e
I20260925 04:34:04.576627 4392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: loading cluster ID for follower catalog manager: success
I20260925 04:34:04.578989 4392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: acquiring CA information for follower catalog manager: success
I20260925 04:34:04.580313 4392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:34:04.580441 4386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:34:04.581110 4386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:34:04.581539 4386 catalog_manager.cc:1259] Loaded cluster ID: c23f755036a34e8aba1ed1cebdb8dd7e
I20260925 04:34:04.581591 4386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:34:04.582065 4386 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:34:04.582439 4386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aab286e68497465588efb5867d9c8536: Loaded TSK: 0
I20260925 04:34:04.582971 4386 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:34:04.656389 4375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:04.656621 4375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:04.656659 4375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:04.660637 4375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:04.660693 4375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:04.660770 4375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:34:04.666184 4375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.193.keytab
--principal=oryx/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:44621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=44163
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:04.667161 4375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:04.668126 4375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:04.678025 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.677992 4402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.678351 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:04.678351 4375 server_base.cc:1034] running on GCE node
I20260925 04:34:04.678894 4375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:04.679507 4375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:04.680685 4375 hybrid_clock.cc:648] HybridClock initialized: now 1790310844680654 us; error 49 us; skew 500 ppm
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:04.683385 4375 init.cc:377] Logged in from keytab as oryx/127.27.46.193@KRBTEST.COM (short username oryx)
I20260925 04:34:04.684340 4375 webserver.cc:460] Webserver started at http://127.27.46.193:44163/ using document root <none> and password file <none>
I20260925 04:34:04.684891 4375 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:04.684962 4375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:04.689234 4375 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:34:04.691438 4409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.692776 4375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:34:04.692916 4375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "a6dfcf7c78694a559e0f7593568e3824"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "35d2f817a574031f01139c5f34c95752"
server_key_iv: "7a4ee5f3abd37b8726fec983161dfa73"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.693632 4375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:04.708377 4375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:04.712000 4375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:04.712219 4375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:04.712909 4375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:04.713882 4375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:04.713943 4375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.713995 4375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:04.714025 4375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.722894 4375 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44621
I20260925 04:34:04.722950 4523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44621 every 8 connection(s)
I20260925 04:34:04.723912 4375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:34:04.731295 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4375
I20260925 04:34:04.731818 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:37745
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:38719
--webserver_port=39597
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:34:04.736888 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.725901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:39185 (local address 127.27.46.254:38719)
0925 04:34:04.726352 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.726372 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.726679 (+ 307us) server_negotiation.cc:407] Connection header received
0925 04:34:04.728067 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.728078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.728365 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.728621 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.729598 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.730540 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.731703 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.732455 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.732656 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.736147 (+ 3491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:39185: BlockingRecv error: recv got EOF from 127.27.46.193:39185 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":97,"threads_started":1}
I20260925 04:34:04.736905 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.726019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:39185)
0925 04:34:04.726501 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.726542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.727716 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.728694 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.728699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.729012 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.729425 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.729437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.730663 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.730665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.731588 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.731597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.732605 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.732608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.732744 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.733340 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.733363 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.735912 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":155,"threads_started":1}
W20260925 04:34:04.737716 4524 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38719 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.46.254:38719: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:04.743769 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.738250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:37553)
0925 04:34:04.738372 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.738391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.738461 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.738726 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.738741 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.738881 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.738978 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.738982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.740022 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.740024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.740568 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.740575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.741217 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.741219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.741284 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.741808 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.741820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.743677 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 04:34:04.743943 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.738311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:37553 (local address 127.27.46.254:38719)
0925 04:34:04.738493 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.738499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.738516 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:34:04.738560 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.738562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.738619 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.738716 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.739092 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.739913 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.740684 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.741103 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.741232 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.743834 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:37553: BlockingRecv error: recv got EOF from 127.27.46.193:37553 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:04.748998 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.744283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:60775)
0925 04:34:04.744454 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.744468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.744549 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.744793 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.744796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.744951 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.745029 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.745033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.745731 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.745734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.746200 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.746208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.746635 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.746639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.746716 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.747151 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.747162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.748907 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 04:34:04.749194 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.744355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60775 (local address 127.27.46.254:38719)
0925 04:34:04.744471 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.744476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.744510 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:34:04.744643 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.744646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.744689 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.744778 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.745130 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.745637 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.746310 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.746526 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.746621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.749083 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:60775: BlockingRecv error: recv got EOF from 127.27.46.193:60775 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 04:34:04.749229 4524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:34:04.843783 4528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:04.844038 4528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:04.844080 4528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:04.848071 4528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:04.848125 4528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:04.848182 4528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:34:04.853255 4528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.194.keytab
--principal=oryx/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:37745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=39597
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:04.854327 4528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:04.855305 4528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:04.864681 4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.864717 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:04.864712 4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:04.865038 4528 server_base.cc:1034] running on GCE node
I20260925 04:34:04.865406 4528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:04.866099 4528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:04.867274 4528 hybrid_clock.cc:648] HybridClock initialized: now 1790310844867259 us; error 26 us; skew 500 ppm
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:04.870066 4528 init.cc:377] Logged in from keytab as oryx/127.27.46.194@KRBTEST.COM (short username oryx)
I20260925 04:34:04.870954 4528 webserver.cc:460] Webserver started at http://127.27.46.194:39597/ using document root <none> and password file <none>
I20260925 04:34:04.871505 4528 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:04.871582 4528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:04.875417 4528 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 04:34:04.877377 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.878592 4528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:34:04.878681 4528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c5339008b5d54d6589dd6610f2b7eac8"
format_stamp: "Formatted at 2026-09-25 04:34:03 on dist-test-slave-r8r3"
server_key: "d8dda7d493313705da98f0bffbeb71d9"
server_key_iv: "6b4f1225b03d11617e0d075557a00e5c"
server_key_version: "encryptionkey@0"
I20260925 04:34:04.879161 4528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:04.903743 4528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:04.906986 4528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:04.907141 4528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:04.907673 4528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:04.908478 4528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:04.908524 4528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.908569 4528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:04.908586 4528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:04.917640 4528 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37745
I20260925 04:34:04.917726 4657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37745 every 8 connection(s)
I20260925 04:34:04.918691 4528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:04.928867 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4528
W20260925 04:34:04.929069 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.920437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:53807 (local address 127.27.46.254:38719)
0925 04:34:04.920580 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.920584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.921381 (+ 797us) server_negotiation.cc:407] Connection header received
0925 04:34:04.922702 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.922705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.922754 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.922827 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.923926 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.924440 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.925551 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.925773 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.925852 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.928910 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:53807: BlockingRecv error: recv got EOF from 127.27.46.194:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:34:04.929359 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:38229
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:38719
--webserver_port=37535
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:34:04.929821 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.920810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:53807)
0925 04:34:04.921235 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.921275 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.922511 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.922986 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.922992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.923330 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.923797 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.923810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.924548 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.924551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.925409 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.925417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.925899 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.925901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.926048 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.926564 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.926579 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.928796 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
W20260925 04:34:04.930270 4658 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38719 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.46.254:38719: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:04.935998 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.930760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:48673)
0925 04:34:04.930940 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.930952 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.931015 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.931265 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.931267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.931387 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.931471 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.931475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.932445 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.932447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.932962 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.932968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.933588 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.933590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.933655 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.934099 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.934110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.935835 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 04:34:04.936017 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.930860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:48673 (local address 127.27.46.254:38719)
0925 04:34:04.930989 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.930995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.931012 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:34:04.931097 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.931102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.931155 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.931234 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.931584 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.932344 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.933083 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.933452 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.933605 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.935934 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:48673: BlockingRecv error: recv got EOF from 127.27.46.194:48673 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:04 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:04.941143 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:04.936489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:55349)
0925 04:34:04.936621 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:04.936635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:04.936735 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:04.936969 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:04.936972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:04.937122 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:04.937187 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.937191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.937937 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.937942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.938444 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:04.938451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.938876 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.938880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:04.938960 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.939391 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:04.939403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:04.941001 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:34:04.941229 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:04.936556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:55349 (local address 127.27.46.254:38719)
0925 04:34:04.936657 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:04.936681 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:04.936696 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:34:04.936832 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:04.936835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:04.936877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:04.936955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:04.937295 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.937836 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.938559 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:04.938769 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:04.938873 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:04.941117 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:55349: BlockingRecv error: recv got EOF from 127.27.46.194:55349 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 04:34:04.941598 4658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:34:05.039842 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:05.040123 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:05.040194 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:05.044067 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:05.044148 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:05.044234 4661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:34:05.049858 4661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/krb5kdc/oryx_127.27.46.195.keytab
--principal=oryx/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:38229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=37535
--tserver_master_addrs=127.27.46.254:38719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:05.050906 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:05.051937 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:05.061189 4666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:05.061209 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:05.061419 4661 server_base.cc:1034] running on GCE node
W20260925 04:34:05.061681 4669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:05.062057 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:05.062618 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:05.063771 4661 hybrid_clock.cc:648] HybridClock initialized: now 1790310845063740 us; error 41 us; skew 500 ppm
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310845, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:05.066905 4661 init.cc:377] Logged in from keytab as oryx/127.27.46.195@KRBTEST.COM (short username oryx)
I20260925 04:34:05.067858 4661 webserver.cc:460] Webserver started at http://127.27.46.195:37535/ using document root <none> and password file <none>
I20260925 04:34:05.068378 4661 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:05.068444 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:05.072456 4661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:34:05.074731 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:05.075896 4661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:34:05.075986 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "2149ed04cd58453a9ceda840dd9c7680"
format_stamp: "Formatted at 2026-09-25 04:34:04 on dist-test-slave-r8r3"
server_key: "2178c493383e5a13883303b55919e166"
server_key_iv: "c0389e8169372c884eb8b252b1afb026"
server_key_version: "encryptionkey@0"
I20260925 04:34:05.076478 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:05.107336 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:05.110735 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:05.110900 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:05.111471 4661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:05.112267 4661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:05.112329 4661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:05.112370 4661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:05.112389 4661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:05.121009 4661 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:38229
I20260925 04:34:05.121032 4790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:38229 every 8 connection(s)
I20260925 04:34:05.122318 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:34:05.126425 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4661
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:34:05.134687 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:05.124232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:60969 (local address 127.27.46.254:38719)
0925 04:34:05.124388 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.124393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.125305 (+ 912us) server_negotiation.cc:407] Connection header received
0925 04:34:05.126700 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.126705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.126763 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.126864 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.128428 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.129157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.130770 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.131049 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.131121 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.134530 (+ 3409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:60969: BlockingRecv error: recv got EOF from 127.27.46.195:60969 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:34:05.135376 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.124688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:60969)
0925 04:34:05.125147 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:05.125190 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:05.126456 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:05.127063 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:05.127070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:05.127528 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:05.128269 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.128292 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.129278 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.129280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.130667 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.130676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.131204 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.131209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.131355 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.131851 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:05.131874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:05.134436 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
W20260925 04:34:05.135850 4791 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38719 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.46.254:38719: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
I20260925 04:34:05.136842 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.127447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.27.46.254:38719)
0925 04:34:05.127653 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.127656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.127667 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:34:05.127701 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.127703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.127737 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.127811 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.128681 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.129341 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.130552 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.130812 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.130904 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.132456 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:05.132462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:05.132475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:05.132499 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:05.134510 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:05.135030 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:05.135036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:05.135040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:05.135161 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:05.135443 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:05.135446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:05.135448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:05.135939 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:05.136143 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:05.136435 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 04:34:05.136700 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:05.141489 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.136320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:47831)
0925 04:34:05.136428 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:05.136442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:05.136524 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:05.136864 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:05.136868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:05.137023 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:05.137117 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.137122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.137997 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.138000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.138765 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.138774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.139239 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.139242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.139332 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.139762 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:05.139775 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:05.141358 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 04:34:05.141613 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:05.136507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:47831 (local address 127.27.46.254:38719)
0925 04:34:05.136672 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.136676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.136688 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:34:05.136725 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.136727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.136757 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.136813 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.137216 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.137902 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.138876 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.139140 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.139227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.141460 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:47831: BlockingRecv error: recv got EOF from 127.27.46.195:47831 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:05.146526 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.142056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:53919)
0925 04:34:05.142145 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:05.142155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:05.142236 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:05.142458 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:05.142461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:05.142595 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:05.142656 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.142661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.143315 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.143320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.143839 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.143847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.144283 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.144286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.144362 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.144806 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:05.144817 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:05.146399 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 04:34:05.146611 4526 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:05.142130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:53919 (local address 127.27.46.254:38719)
0925 04:34:05.142267 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.142272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.142286 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:34:05.142317 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.142319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.142349 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.142444 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.142759 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.143216 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.143963 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.144180 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.144280 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.146499 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:53919: BlockingRecv error: recv got EOF from 127.27.46.195:53919 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 04:34:05.146819 4791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:05.755215 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.749839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:36293)
0925 04:34:05.750120 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:05.750133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:05.750225 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:05.750548 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:05.750551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:05.750787 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:05.750915 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.750928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.751810 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.751815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.752355 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.752363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.752812 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.752816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.752901 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.753364 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:05.753375 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:05.755103 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260925 04:34:05.755379 4798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:05.750041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:36293 (local address 127.27.46.254:38719)
0925 04:34:05.750285 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.750288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.750300 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:34:05.750336 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.750338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.750396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.750525 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.751043 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.751686 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.752466 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.752700 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.752805 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.755224 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:36293: BlockingRecv error: recv got EOF from 127.27.46.193:36293 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:05 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:05.947449 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:05.942220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:37631)
0925 04:34:05.942465 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:05.942478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:05.942568 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:05.942773 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:05.942775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:05.942942 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:05.943032 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.943037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.943752 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.943754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.944326 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:05.944333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.944718 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.944720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:05.944780 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.945341 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:05.945352 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:05.947270 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260925 04:34:05.947492 4798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:05.942323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:37631 (local address 127.27.46.254:38719)
0925 04:34:05.942503 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:05.942507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:05.942524 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:34:05.942654 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:05.942656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:05.942698 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:05.942763 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:05.943135 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.943663 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.944419 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:05.944629 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:05.944701 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:05.947391 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:37631: BlockingRecv error: recv got EOF from 127.27.46.194:37631 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:06.152438 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:06.147441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:37853)
0925 04:34:06.147748 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:06.147761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:06.147856 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:06.148082 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:06.148084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:06.148253 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:06.148347 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.148353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.149046 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.149048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.149673 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.149683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.150065 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.150068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.150124 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.150574 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:06.150585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:06.152318 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":110,"threads_started":1}
W20260925 04:34:06.152562 4798 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:06.147528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:37853 (local address 127.27.46.254:38719)
0925 04:34:06.147679 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:06.147683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:06.147826 (+ 143us) server_negotiation.cc:407] Connection header received
0925 04:34:06.147938 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:06.147940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:06.147987 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:06.148046 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:06.148443 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.148951 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.149769 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.149976 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.150051 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.152447 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:37853: BlockingRecv error: recv got EOF from 127.27.46.195:37853 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:06.761332 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:06.756016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:60839)
0925 04:34:06.756301 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:06.756314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:06.756412 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:06.756615 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:06.756616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:06.756789 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:06.756885 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.756889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.757617 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.757620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.758156 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.758164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.758623 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.758626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.758686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.759260 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:06.759273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:06.761157 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
W20260925 04:34:06.761420 4802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:06.756054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60839 (local address 127.27.46.254:38719)
0925 04:34:06.756291 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:06.756294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:06.756381 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:34:06.756485 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:06.756487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:06.756530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:06.756605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:06.756976 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.757529 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.758278 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.758505 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.758605 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.761291 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:60839: BlockingRecv error: recv got EOF from 127.27.46.193:60839 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:06 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:06.954010 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:06.948525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:52523)
0925 04:34:06.948794 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:06.948807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:06.948903 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:06.949140 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:06.949141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:06.949345 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:06.949436 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.949440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.950231 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.950234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.951064 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:06.951077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.951557 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.951560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:06.951623 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.952142 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:06.952154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:06.953890 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
W20260925 04:34:06.954172 4802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:06.948579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52523 (local address 127.27.46.254:38719)
0925 04:34:06.948817 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:06.948821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:06.948869 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:34:06.949013 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:06.949015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:06.949057 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:06.949124 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:06.949534 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.950126 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.951189 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:06.951459 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:06.951541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:06.954023 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:52523: BlockingRecv error: recv got EOF from 127.27.46.194:52523 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:07.163817 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:07.153202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:52611)
0925 04:34:07.153552 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:07.153569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:07.153730 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:07.154014 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:07.154018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:07.154221 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:07.154315 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.154321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.155073 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.155078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.155641 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.155648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.156097 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.156100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.156188 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.161756 (+ 5568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:07.161769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:07.163644 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
W20260925 04:34:07.163877 4802 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:07.153359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52611 (local address 127.27.46.254:38719)
0925 04:34:07.153547 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:07.153553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:07.153677 (+ 124us) server_negotiation.cc:407] Connection header received
0925 04:34:07.153839 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:07.153842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:07.153894 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:07.153981 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:07.154438 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.154962 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.155760 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.155983 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.156093 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.163766 (+ 7673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:52611: BlockingRecv error: recv got EOF from 127.27.46.195:52611 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:07.767457 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:07.762289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:54705)
0925 04:34:07.762593 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:07.762625 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:07.762714 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:07.762959 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:07.762962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:07.763138 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:07.763236 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.763241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.763968 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.763971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.764516 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.764523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.764922 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.764925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.764984 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.765439 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:07.765450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:07.767320 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
W20260925 04:34:07.767621 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:07.762357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:54705 (local address 127.27.46.254:38719)
0925 04:34:07.762672 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:07.762675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:07.762689 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:34:07.762802 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:07.762804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:07.762849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:07.762924 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:07.763334 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.763873 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.764614 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.764825 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.764898 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.767469 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:54705: BlockingRecv error: recv got EOF from 127.27.46.193:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":84,"threads_started":1}
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:07 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:07.960361 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:07.954742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:50317)
0925 04:34:07.955044 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:07.955066 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:07.955152 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:07.955392 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:07.955394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:07.955575 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:07.955674 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.955679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.956678 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.956681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.957325 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:07.957335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.957760 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.957763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:07.957824 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.958406 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:07.958419 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:07.960223 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
W20260925 04:34:07.960513 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:07.954972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:50317 (local address 127.27.46.254:38719)
0925 04:34:07.955111 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:07.955114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:07.955128 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:34:07.955234 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:07.955236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:07.955279 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:07.955351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:07.955791 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.956567 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.957431 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:07.957666 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:07.957745 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:07.960354 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:50317: BlockingRecv error: recv got EOF from 127.27.46.194:50317 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:08.170450 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:08.164718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:43535)
0925 04:34:08.165029 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:08.165043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:08.165152 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:08.165392 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:08.165394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:08.165630 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:08.165733 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.165739 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.166451 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.166454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.167035 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.167042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.167640 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.167644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.167741 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.168425 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:08.168440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:08.170307 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
W20260925 04:34:08.170606 4806 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:08.164794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:43535 (local address 127.27.46.254:38719)
0925 04:34:08.164954 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:08.164961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:08.165121 (+ 160us) server_negotiation.cc:407] Connection header received
0925 04:34:08.165236 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:08.165238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:08.165288 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:08.165349 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:08.165843 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.166357 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.167159 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.167468 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.167598 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.170450 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:43535: BlockingRecv error: recv got EOF from 127.27.46.195:43535 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:08.774046 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:08.768386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:60701)
0925 04:34:08.768677 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:08.768691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:08.768812 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:08.769032 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:08.769034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:08.769208 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:08.769304 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.769309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.770045 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.770048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.770630 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.770639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.771064 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.771067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.771127 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.771757 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:08.771770 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:08.773867 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
W20260925 04:34:08.774153 4810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:08.768452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60701 (local address 127.27.46.254:38719)
0925 04:34:08.768736 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:08.768740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:08.768771 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:34:08.768898 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:08.768900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:08.768945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:08.769022 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:08.769398 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.769940 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.770731 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.770961 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.771042 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.774005 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:60701: BlockingRecv error: recv got EOF from 127.27.46.193:60701 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:08 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:08.967144 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:08.961188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:48895)
0925 04:34:08.961683 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:08.961697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:08.961852 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:08.962172 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:08.962175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:08.962402 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:08.962502 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.962508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.963317 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.963321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.963898 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:08.963906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.964387 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.964391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:08.964479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.965050 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:08.965063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:08.967009 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":140,"threads_started":1}
W20260925 04:34:08.967278 4810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:08.961351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:48895 (local address 127.27.46.254:38719)
0925 04:34:08.961628 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:08.961634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:08.961784 (+ 150us) server_negotiation.cc:407] Connection header received
0925 04:34:08.961978 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:08.961991 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:08.962038 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:08.962135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:08.962640 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.963191 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.964026 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:08.964254 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:08.964370 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:08.967155 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:48895: BlockingRecv error: recv got EOF from 127.27.46.194:48895 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:09.177466 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:09.171504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:37379)
0925 04:34:09.171818 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:09.171830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:09.171968 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:09.172410 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:09.172417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:09.172660 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:09.172817 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.172825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.173672 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.173677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.174269 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.174277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.174780 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.174783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.174869 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.175447 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:09.175468 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:09.177344 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
W20260925 04:34:09.177681 4810 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:09.171597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:37379 (local address 127.27.46.254:38719)
0925 04:34:09.171730 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:09.171735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:09.171906 (+ 171us) server_negotiation.cc:407] Connection header received
0925 04:34:09.172152 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:09.172156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:09.172216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:09.172298 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:09.172940 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.173560 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.174393 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.174671 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.174774 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.177490 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:37379: BlockingRecv error: recv got EOF from 127.27.46.195:37379 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:09.780788 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:09.775016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:37711)
0925 04:34:09.775303 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:09.775317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:09.775414 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:09.775645 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:09.775647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:09.775829 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:09.775968 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.775979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.776905 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.776908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.777678 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.777688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.778111 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.778114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.778171 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.778718 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:09.778731 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:09.780672 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
W20260925 04:34:09.781006 4814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:09.775044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:37711 (local address 127.27.46.254:38719)
0925 04:34:09.775314 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:09.775318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:09.775392 (+ 74us) server_negotiation.cc:407] Connection header received
0925 04:34:09.775482 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:09.775484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:09.775526 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:09.775602 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:09.776103 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.776796 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.777778 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.778012 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.778095 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.780858 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:37711: BlockingRecv error: recv got EOF from 127.27.46.193:37711 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:09 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:09.974074 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:09.968013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:41067)
0925 04:34:09.968387 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:09.968403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:09.968548 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:09.968831 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:09.968834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:09.969060 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:09.969155 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.969160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.969986 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.969991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.970625 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:09.970635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.971106 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.971109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:09.971190 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.971755 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:09.971767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:09.973941 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
W20260925 04:34:09.974220 4814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:09.968205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:41067 (local address 127.27.46.254:38719)
0925 04:34:09.968389 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:09.968395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:09.968495 (+ 100us) server_negotiation.cc:407] Connection header received
0925 04:34:09.968658 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:09.968663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:09.968717 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:09.968807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:09.969278 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.969873 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.970761 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:09.970986 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:09.971120 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:09.974072 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:41067: BlockingRecv error: recv got EOF from 127.27.46.194:41067 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:10.184011 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:10.178304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:37017)
0925 04:34:10.178556 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:10.178568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:10.178653 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:10.178859 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:10.178861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:10.179023 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:10.179122 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.179128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.179895 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.179898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.180496 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.180505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.181091 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.181094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.181160 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.181836 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:10.181852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:10.183896 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260925 04:34:10.184330 4814 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:10.178336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:37017 (local address 127.27.46.254:38719)
0925 04:34:10.178460 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:10.178465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:10.178630 (+ 165us) server_negotiation.cc:407] Connection header received
0925 04:34:10.178723 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:10.178725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:10.178767 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:10.178826 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:10.179224 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.179807 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.180774 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.180989 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.181074 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.184033 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:37017: BlockingRecv error: recv got EOF from 127.27.46.195:37017 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:10.787256 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:10.781806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:43697)
0925 04:34:10.782111 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:10.782126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:10.782216 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:10.782625 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:10.782628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:10.782806 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:10.782966 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.782973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.783716 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.783718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.784260 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.784268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.784691 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.784694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.784751 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.785211 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:10.785222 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:10.787138 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260925 04:34:10.787449 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:10.781865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:43697 (local address 127.27.46.254:38719)
0925 04:34:10.782176 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:10.782180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:10.782193 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:34:10.782419 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:10.782424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:10.782503 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:10.782587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:10.783072 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.783616 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.784358 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.784596 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.784673 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.787300 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:43697: BlockingRecv error: recv got EOF from 127.27.46.193:43697 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:10 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:10.980163 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:10.974869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:37427)
0925 04:34:10.975157 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:10.975170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:10.975267 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:10.975489 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:10.975491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:10.975681 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:10.975787 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.975792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.976493 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.976495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.977030 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:10.977037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.977422 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.977442 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:10.977504 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.978029 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:10.978040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:10.979985 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
W20260925 04:34:10.980273 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:10.974947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:37427 (local address 127.27.46.254:38719)
0925 04:34:10.975100 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:10.975107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:10.975238 (+ 131us) server_negotiation.cc:407] Connection header received
0925 04:34:10.975343 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:10.975345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:10.975389 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:10.975455 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:10.975881 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.976406 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.977127 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:10.977335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:10.977403 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:10.980126 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:37427: BlockingRecv error: recv got EOF from 127.27.46.194:37427 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:11.189937 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:11.184729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:33873)
0925 04:34:11.184997 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:11.185010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:11.185105 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:11.185314 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:11.185316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:11.185584 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:11.185691 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.185697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.186467 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.186469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.187003 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.187010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.187423 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.187426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.187487 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.187964 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:11.187975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:11.189820 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260925 04:34:11.190126 4818 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:11.184833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:33873 (local address 127.27.46.254:38719)
0925 04:34:11.184961 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:11.184965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:11.185074 (+ 109us) server_negotiation.cc:407] Connection header received
0925 04:34:11.185171 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:11.185173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:11.185216 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:11.185284 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:11.185796 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.186362 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.187113 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.187327 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.187402 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.189956 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:33873: BlockingRecv error: recv got EOF from 127.27.46.195:33873 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:11.793852 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:11.788021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:57605)
0925 04:34:11.788358 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:11.788372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:11.788467 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:11.788776 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:11.788779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:11.788993 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:11.789095 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.789102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.790055 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.790060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.790727 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.790737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.791167 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.791169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.791228 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.791731 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:11.791743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:11.793675 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
W20260925 04:34:11.793936 4822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:11.788220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57605 (local address 127.27.46.254:38719)
0925 04:34:11.788491 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:11.788495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:11.788510 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:34:11.788568 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:11.788570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:11.788615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:11.788721 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:11.789212 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.789929 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.790837 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.791070 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.791152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.793807 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:57605: BlockingRecv error: recv got EOF from 127.27.46.193:57605 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:11 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:11.987051 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:11.981108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:51147)
0925 04:34:11.981441 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:11.981462 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:11.981601 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:11.981844 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:11.981846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:11.982108 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:11.982215 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.982223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.982971 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.982974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.983756 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:11.983767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.984208 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.984211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:11.984277 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.984793 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:11.984806 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:11.986875 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":97,"threads_started":1}
W20260925 04:34:11.987131 4822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:11.981190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:51147 (local address 127.27.46.254:38719)
0925 04:34:11.981333 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:11.981336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:11.981571 (+ 235us) server_negotiation.cc:407] Connection header received
0925 04:34:11.981684 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:11.981686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:11.981733 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:11.981804 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:11.982341 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.982866 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.983875 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:11.984102 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:11.984182 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:11.987000 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:51147: BlockingRecv error: recv got EOF from 127.27.46.194:51147 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:12.199702 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:12.190654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:40755)
0925 04:34:12.190989 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:12.191003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:12.191119 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:12.191350 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:12.191352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:12.191537 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:12.191640 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.191646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.192360 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.192363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.192918 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.192925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.193327 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.193330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.193389 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.197731 (+ 4342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:12.197743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:12.199579 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
W20260925 04:34:12.199841 4822 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:12.190741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:40755 (local address 127.27.46.254:38719)
0925 04:34:12.190913 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:12.190918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:12.191078 (+ 160us) server_negotiation.cc:407] Connection header received
0925 04:34:12.191199 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:12.191201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:12.191248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:12.191306 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:12.191744 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.192262 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.193019 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.193229 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.193303 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.199722 (+ 6419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:40755: BlockingRecv error: recv got EOF from 127.27.46.195:40755 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:12.800513 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:12.794893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:57247)
0925 04:34:12.795193 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:12.795230 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:12.795352 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:12.795618 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:12.795620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:12.795801 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:12.795905 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.795910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.796664 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.796666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.797301 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.797311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.797845 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.797847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.797909 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.798519 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:12.798532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:12.800389 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
W20260925 04:34:12.800721 4826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:12.795008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:57247 (local address 127.27.46.254:38719)
0925 04:34:12.795263 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:12.795268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:12.795284 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:34:12.795451 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:12.795453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:12.795499 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:12.795583 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:12.796004 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.796566 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.797438 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.797739 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.797822 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.800536 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:57247: BlockingRecv error: recv got EOF from 127.27.46.193:57247 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:12 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:12.993321 4827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:12.987932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:52323)
0925 04:34:12.988234 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:12.988248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:12.988352 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:12.988596 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:12.988598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:12.988780 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:12.988877 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.988881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.989782 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.989784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.990338 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:12.990346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.990756 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.990758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:12.990823 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.991335 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:12.991348 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:12.993196 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
W20260925 04:34:12.993503 4826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:12.988001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52323 (local address 127.27.46.254:38719)
0925 04:34:12.988159 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:12.988165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:12.988323 (+ 158us) server_negotiation.cc:407] Connection header received
0925 04:34:12.988437 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:12.988439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:12.988485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:12.988545 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:12.988985 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.989681 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.990443 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:12.990655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:12.990730 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:12.993366 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:52323: BlockingRecv error: recv got EOF from 127.27.46.194:52323 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:13.207273 4828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:13.200842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:44533)
0925 04:34:13.201285 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:13.201309 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:13.201463 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:13.201808 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:13.201812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:13.202042 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:13.202144 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.202149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.202933 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.202937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.203661 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.203673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.204286 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.204289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.204397 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:13.205087 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:13.205100 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:13.207145 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":122,"threads_started":1}
W20260925 04:34:13.207490 4826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:13.201015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:44533 (local address 127.27.46.254:38719)
0925 04:34:13.201248 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:13.201254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:13.201396 (+ 142us) server_negotiation.cc:407] Connection header received
0925 04:34:13.201621 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:13.201624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:13.201684 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:13.201778 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:13.202274 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.202807 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.203815 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.204146 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.204267 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:13.207336 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:44533: BlockingRecv error: recv got EOF from 127.27.46.195:44533 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:13.807183 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:13.801286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:53785)
0925 04:34:13.801686 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:13.801703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:13.801845 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:13.802152 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:13.802155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:13.802374 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:13.802486 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.802492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.803328 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.803333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.803920 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.803928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.804416 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.804420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.804504 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:13.805078 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:13.805091 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:13.807002 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":90,"threads_started":1}
W20260925 04:34:13.807262 4830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:13.801414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:53785 (local address 127.27.46.254:38719)
0925 04:34:13.801686 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:13.801692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:13.801786 (+ 94us) server_negotiation.cc:407] Connection header received
0925 04:34:13.801966 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:13.801969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:13.802028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:13.802126 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:13.802621 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.803211 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.804046 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.804290 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.804400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:13.807133 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:53785: BlockingRecv error: recv got EOF from 127.27.46.193:53785 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:13 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:14.000330 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:13.994238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:48157)
0925 04:34:13.994604 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:13.994617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:13.994761 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:13.995052 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:13.995055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:13.995270 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:13.995376 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.995381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.996167 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.996172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.996812 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:13.996821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.997358 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.997361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:13.997426 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:13.998089 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:13.998102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:14.000201 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
W20260925 04:34:14.000509 4830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:13.994374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:48157 (local address 127.27.46.254:38719)
0925 04:34:13.994551 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:13.994557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:13.994694 (+ 137us) server_negotiation.cc:407] Connection header received
0925 04:34:13.994881 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:13.994884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:13.994938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:13.995028 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:13.995495 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.996052 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.996991 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:13.997225 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:13.997330 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:14.000344 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:48157: BlockingRecv error: recv got EOF from 127.27.46.194:48157 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 04:34:14 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:14 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:14.213829 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:14.208179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:42411)
0925 04:34:14.208472 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:14.208485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:14.208588 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:14.208815 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:14.208817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:14.208996 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:14.209112 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:14.209118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.209876 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.209878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:14.210430 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:14.210437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.210914 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.210917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:14.210976 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:14.211544 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:14.211556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:14.213638 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260925 04:34:14.213910 4830 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:14.208283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:42411 (local address 127.27.46.254:38719)
0925 04:34:14.208432 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:14.208436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:14.208537 (+ 101us) server_negotiation.cc:407] Connection header received
0925 04:34:14.208658 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:14.208660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:14.208705 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:14.208777 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:14.209222 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.209774 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.210567 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.210797 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.210907 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:14.213792 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:42411: BlockingRecv error: recv got EOF from 127.27.46.195:42411 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 04:34:14 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:14 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:14.813402 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:14.808074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:34473)
0925 04:34:14.808377 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:14.808390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:14.808469 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:14.808689 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:14.808691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:14.808865 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:14.808959 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:14.808964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.809799 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.809801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:14.810364 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:14.810372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.810791 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.810794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:14.810855 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:14.811430 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:14.811442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:14.813281 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":122,"threads_started":1}
W20260925 04:34:14.813632 4834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:14.808210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:34473 (local address 127.27.46.254:38719)
0925 04:34:14.808458 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:14.808462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:14.808474 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:34:14.808535 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:14.808537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:14.808578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:14.808654 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:14.809081 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.809692 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.810467 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:14.810689 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:14.810764 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:14.813424 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:34473: BlockingRecv error: recv got EOF from 127.27.46.193:34473 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
Sep 25 04:34:14 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:15 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:15.006546 4835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:15.001162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:45205)
0925 04:34:15.001544 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:15.001561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:15.001673 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:15.001880 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:15.001882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:15.002066 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:15.002161 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.002165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.002863 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.002865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.003415 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.003422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.003874 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.003877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.003946 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.004417 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:15.004429 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:15.006428 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
W20260925 04:34:15.006729 4834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:15.001218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:45205 (local address 127.27.46.254:38719)
0925 04:34:15.001338 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:15.001342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:15.001624 (+ 282us) server_negotiation.cc:407] Connection header received
0925 04:34:15.001742 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:15.001744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:15.001790 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:15.001854 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:15.002256 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.002774 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.003512 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.003734 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.003811 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.006570 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:45205: BlockingRecv error: recv got EOF from 127.27.46.194:45205 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 04:34:15 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:15 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:15.220208 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:15.214618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:38085)
0925 04:34:15.214950 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:15.214967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:15.215107 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:15.215396 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:15.215399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:15.215612 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:15.215708 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.215714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.216491 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.216496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.217071 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.217078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.217598 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.217602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.217692 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.218168 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:15.218179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:15.220014 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":125,"threads_started":1}
W20260925 04:34:15.220263 4834 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:15.214781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:38085 (local address 127.27.46.254:38719)
0925 04:34:15.214936 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:15.214941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:15.215049 (+ 108us) server_negotiation.cc:407] Connection header received
0925 04:34:15.215216 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:15.215220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:15.215270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:15.215367 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:15.215833 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.216376 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.217192 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.217434 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.217586 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.220153 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:38085: BlockingRecv error: recv got EOF from 127.27.46.195:38085 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 04:34:15 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:15 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:15.819550 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:15.814210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:59407)
0925 04:34:15.814501 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:15.814516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:15.814614 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:15.814866 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:15.814868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:15.815052 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:15.815149 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.815154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.815906 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.815909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.816472 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:15.816481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.816901 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.816904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:15.816964 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.817449 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:15.817461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:15.819402 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
W20260925 04:34:15.819747 4838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:15.814301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:59407 (local address 127.27.46.254:38719)
0925 04:34:15.814575 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:15.814578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:15.814592 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:34:15.814683 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:15.814685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:15.814727 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:15.814801 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:15.815253 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.815783 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.816582 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:15.816799 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:15.816873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:15.819566 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:59407: BlockingRecv error: recv got EOF from 127.27.46.193:59407 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:16.013161 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:16.007344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:52913)
0925 04:34:16.007801 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:16.007815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:16.007959 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:16.008243 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:16.008246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:16.008454 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:16.008592 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.008600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.009643 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.009647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.010323 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.010332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.010760 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.010763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.010824 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.011308 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:16.011320 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:16.013022 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":90,"threads_started":1}
W20260925 04:34:16.013281 4838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:16.007490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52913 (local address 127.27.46.254:38719)
0925 04:34:16.007755 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:16.007760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:16.007891 (+ 131us) server_negotiation.cc:407] Connection header received
0925 04:34:16.008070 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:16.008073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:16.008127 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:16.008218 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:16.008745 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.009542 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.010436 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.010658 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.010741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.013160 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:52913: BlockingRecv error: recv got EOF from 127.27.46.194:52913 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:16.226935 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:16.221147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:34055)
0925 04:34:16.221410 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:16.221433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:16.221598 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:16.221879 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:16.221882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:16.222094 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:16.222192 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.222198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.222972 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.222976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.223525 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.223533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.224053 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.224056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.224138 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.224727 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:16.224740 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:16.226735 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
W20260925 04:34:16.226984 4838 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:16.221247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:34055 (local address 127.27.46.254:38719)
0925 04:34:16.221387 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:16.221392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:16.221514 (+ 122us) server_negotiation.cc:407] Connection header received
0925 04:34:16.221703 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:16.221706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:16.221758 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:16.221853 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:16.222319 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.222861 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.223692 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.223935 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.224057 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.226875 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:34055: BlockingRecv error: recv got EOF from 127.27.46.195:34055 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:16 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:16.825745 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:16.820270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:56387)
0925 04:34:16.820552 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:16.820565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:16.820659 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:16.820937 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:16.820939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:16.821111 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:16.821208 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.821214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.822010 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.822013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.822592 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:16.822601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.823056 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.823059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:16.823122 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.823647 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:16.823659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:16.825626 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260925 04:34:16.825923 4842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:16.820373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:56387 (local address 127.27.46.254:38719)
0925 04:34:16.820611 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:16.820615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:16.820628 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:34:16.820743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:16.820745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:16.820789 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:16.820866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:16.821311 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.821874 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.822715 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:16.822948 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:16.823030 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:16.825770 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:56387: BlockingRecv error: recv got EOF from 127.27.46.193:56387 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:17.020255 4843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:17.014209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:47805)
0925 04:34:17.014666 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:17.014679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:17.014832 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:17.015299 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:17.015302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:17.015498 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:17.015651 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.015682 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.016576 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.016579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.017127 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.017135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.017584 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.017586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.017648 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.018115 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:17.018126 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:17.020140 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":84,"threads_started":1}
W20260925 04:34:17.020392 4842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:17.014343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:47805 (local address 127.27.46.254:38719)
0925 04:34:17.014612 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:17.014617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:17.014773 (+ 156us) server_negotiation.cc:407] Connection header received
0925 04:34:17.015096 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:17.015099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:17.015140 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:17.015247 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:17.015800 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.016477 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.017226 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.017433 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.017525 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.020271 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:47805: BlockingRecv error: recv got EOF from 127.27.46.194:47805 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:17.237251 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:17.227843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:51747)
0925 04:34:17.228145 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:17.228157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:17.228284 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:17.228555 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:17.228558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:17.228767 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:17.228858 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.228863 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.229687 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.229692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.230236 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.230244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.230709 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.230712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.230796 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.235253 (+ 4457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:17.235273 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:17.237066 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
W20260925 04:34:17.237296 4842 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:17.227901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:51747 (local address 127.27.46.254:38719)
0925 04:34:17.228074 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:17.228079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:17.228234 (+ 155us) server_negotiation.cc:407] Connection header received
0925 04:34:17.228390 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:17.228393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:17.228453 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:17.228534 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:17.228980 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.229586 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.230359 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.230586 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.230703 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.237196 (+ 6493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:51747: BlockingRecv error: recv got EOF from 127.27.46.195:51747 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:17 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:17.832101 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:17.826503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:51465)
0925 04:34:17.826812 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:17.826825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:17.826915 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:17.827156 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:17.827158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:17.827331 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:17.827426 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.827432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.828185 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.828187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.828741 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:17.828748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.829308 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.829311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:17.829376 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.829975 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:17.829987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:17.831959 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
W20260925 04:34:17.832232 4846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:17.826602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:51465 (local address 127.27.46.254:38719)
0925 04:34:17.826857 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:17.826861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:17.826874 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:34:17.826997 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:17.826999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:17.827045 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:17.827122 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:17.827539 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.828095 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.828836 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:17.829171 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:17.829291 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:17.832098 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:51465: BlockingRecv error: recv got EOF from 127.27.46.193:51465 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:18.027397 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:18.021269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:60137)
0925 04:34:18.021650 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:18.021668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:18.021789 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:18.022073 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:18.022076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:18.022291 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:18.022383 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.022389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.023238 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.023241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.024119 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.024134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.024726 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.024729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.024791 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.025321 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:18.025332 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:18.027211 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
W20260925 04:34:18.027441 4846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:18.021540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60137 (local address 127.27.46.254:38719)
0925 04:34:18.021676 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:18.021683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:18.021730 (+ 47us) server_negotiation.cc:407] Connection header received
0925 04:34:18.021909 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:18.021912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:18.021966 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:18.022069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:18.022502 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.023065 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.024240 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.024598 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.024693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.027336 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:60137: BlockingRecv error: recv got EOF from 127.27.46.194:60137 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:18.243876 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:18.238148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:48841)
0925 04:34:18.238513 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:18.238525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:18.238679 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:18.238951 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:18.238954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:18.239177 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:18.239273 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.239278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.240042 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.240053 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.240644 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.240652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.241152 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.241156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.241248 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.241860 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:18.241872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:18.243687 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
W20260925 04:34:18.243919 4846 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:18.238273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:48841 (local address 127.27.46.254:38719)
0925 04:34:18.238461 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:18.238466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:18.238596 (+ 130us) server_negotiation.cc:407] Connection header received
0925 04:34:18.238782 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:18.238786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:18.238842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:18.238930 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:18.239392 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.239927 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.240774 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.241002 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.241117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.243809 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:48841: BlockingRecv error: recv got EOF from 127.27.46.195:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:18 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:18.838179 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:18.832932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:60707)
0925 04:34:18.833221 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:18.833234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:18.833331 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:18.833607 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:18.833609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:18.833785 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:18.833888 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.833893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.834613 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.834615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.835213 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:18.835222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.835640 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.835642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:18.835705 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.836191 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:18.836203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:18.838054 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260925 04:34:18.838351 4850 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:18.833055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60707 (local address 127.27.46.254:38719)
0925 04:34:18.833351 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:18.833355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:18.833367 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:34:18.833400 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:18.833402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:18.833440 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:18.833587 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:18.833997 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.834515 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.835317 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:18.835542 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:18.835615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:18.838182 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:60707: BlockingRecv error: recv got EOF from 127.27.46.193:60707 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:19.033836 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:19.028203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:52081)
0925 04:34:19.028532 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:19.028547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:19.028680 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:19.028961 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:19.028964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:19.029172 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:19.029260 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.029265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.030111 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.030116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.030682 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.030689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.031164 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.031167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.031255 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.031837 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:19.031849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:19.033659 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
W20260925 04:34:19.033900 4850 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:19.028318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52081 (local address 127.27.46.254:38719)
0925 04:34:19.028477 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:19.028482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:19.028631 (+ 149us) server_negotiation.cc:407] Connection header received
0925 04:34:19.028771 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:19.028774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:19.028822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:19.028936 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:19.029386 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.029992 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.030801 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.031045 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.031150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.033778 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:52081: BlockingRecv error: recv got EOF from 127.27.46.194:52081 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.195@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:19.250236 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:19.244672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.195:36837)
0925 04:34:19.244966 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:19.244979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:19.245104 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:19.245374 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:19.245377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:19.245617 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:19.245742 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.245748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.246536 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.246542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.247153 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.247161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.247652 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.247656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.247744 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.248200 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:19.248211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:19.250114 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
W20260925 04:34:19.250361 4850 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:19.244730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36837 (local address 127.27.46.254:38719)
0925 04:34:19.244916 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:19.244921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:19.245064 (+ 143us) server_negotiation.cc:407] Connection header received
0925 04:34:19.245208 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:19.245210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:19.245267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:19.245351 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:19.245863 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.246421 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.247275 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.247537 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.247649 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.250246 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.195:36837: BlockingRecv error: recv got EOF from 127.27.46.195:36837 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:19 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.193@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:19.844509 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:19.839052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.193:58337)
0925 04:34:19.839321 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:19.839341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:19.839433 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:19.839687 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:19.839689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:19.839870 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:19.839966 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.839971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.840700 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.840703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.841252 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:19.841260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.841698 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.841700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:19.841761 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.842229 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:19.842241 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:19.844392 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
W20260925 04:34:19.844668 4854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:19.839187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:58337 (local address 127.27.46.254:38719)
0925 04:34:19.839449 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:19.839458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:19.839469 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:34:19.839504 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:19.839506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:19.839550 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:19.839651 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:19.840080 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.840607 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.841354 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:19.841599 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:19.841675 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:19.844530 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.193:58337: BlockingRecv error: recv got EOF from 127.27.46.193:58337 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
Sep 25 04:34:20 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:34:20 dist-test-slave-r8r3 krb5kdc[3786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.46.194@KRBTEST.COM for oryx/127.27.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:34:20.040512 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.034871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38719 (local address 127.27.46.194:37385)
0925 04:34:20.035308 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:20.035325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:20.035454 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:20.035727 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:20.035729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:20.035935 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:20.036022 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.036027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.036855 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.036859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.037420 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.037428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.037910 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.037914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.037997 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.038490 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:20.038501 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:20.040344 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.46.254:38719: Server oryx/127.27.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":113,"threads_started":1}
W20260925 04:34:20.040572 4854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:20.035063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:37385 (local address 127.27.46.254:38719)
0925 04:34:20.035311 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:20.035317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:20.035402 (+ 85us) server_negotiation.cc:407] Connection header received
0925 04:34:20.035562 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:20.035567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:20.035619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:20.035707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:20.036134 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.036745 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.037567 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.037795 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.037908 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.040465 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.46.194:37385: BlockingRecv error: recv got EOF from 127.27.46.194:37385 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260925 04:34:20.132432 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4375
I20260925 04:34:20.138842 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4528
I20260925 04:34:20.144181 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4661
I20260925 04:34:20.149181 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4305
2026-09-25T04:34:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18099 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T04:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:20Z Disabled control of system clock
I20260925 04:34:20.178989 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35387
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:34:20.289629 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:20.289877 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:20.289912 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:20.293941 4859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:20.293992 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:20.294008 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:20.294024 4859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:20.294040 4859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:20.298792 4859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:20.300010 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:20.301318 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:20.310623 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:20.310623 4868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.310796 4859 server_base.cc:1034] running on GCE node
W20260925 04:34:20.310702 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.311415 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:20.312403 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:20.313572 4859 hybrid_clock.cc:648] HybridClock initialized: now 1790310860313558 us; error 24 us; skew 500 ppm
I20260925 04:34:20.315250 4859 webserver.cc:460] Webserver started at http://127.27.46.254:40879/ using document root <none> and password file <none>
I20260925 04:34:20.315810 4859 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:20.315871 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:20.316098 4859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:20.318140 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "9555cf68cd544f65a2f976b0981edb2b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "ed355b23c63d752e16bb4c4e2639cfbd"
server_key_iv: "80b252af49bab73ff68b36f69dd6825e"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.318687 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "9555cf68cd544f65a2f976b0981edb2b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "ed355b23c63d752e16bb4c4e2639cfbd"
server_key_iv: "80b252af49bab73ff68b36f69dd6825e"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.322248 4859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:34:20.324571 4874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.325819 4859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:20.325938 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "9555cf68cd544f65a2f976b0981edb2b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "ed355b23c63d752e16bb4c4e2639cfbd"
server_key_iv: "80b252af49bab73ff68b36f69dd6825e"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.326045 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:20.352314 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:20.353020 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:20.353482 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:20.360268 4859 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35387
I20260925 04:34:20.360275 4926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35387 every 8 connection(s)
I20260925 04:34:20.361215 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:34:20.364327 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4859
I20260925 04:34:20.364462 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:34:20.364480 4927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:20.364780 27835 external_mini_cluster.cc:1428] Setting key c71f7109ec175f043c9166640c13e597
I20260925 04:34:20.370862 4927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b: Bootstrap starting.
I20260925 04:34:20.373350 4927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:20.373797 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.366139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.27.46.254:35387)
0925 04:34:20.366728 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:20.366743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:20.366777 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:34:20.367524 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:20.367541 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:20.367943 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:20.368209 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:20.368508 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.369370 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.370570 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.371119 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.371238 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.372080 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:20.372085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:20.372097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:20.372458 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:20.372509 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:20.372528 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:20.372603 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:20.372750 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:34:20.373048 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":142,"threads_started":1}
I20260925 04:34:20.374223 4927 log.cc:824] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:20.376116 4927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b: No bootstrap required, opened a new log
I20260925 04:34:20.379313 4927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } }
I20260925 04:34:20.379508 4927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:20.379554 4927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9555cf68cd544f65a2f976b0981edb2b, State: Initialized, Role: FOLLOWER
I20260925 04:34:20.379906 4927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } }
I20260925 04:34:20.380012 4927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:34:20.380055 4927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:34:20.380182 4927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:20.381145 4927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } }
I20260925 04:34:20.381414 4927 leader_election.cc:302] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9555cf68cd544f65a2f976b0981edb2b; no voters:
I20260925 04:34:20.381721 4927 leader_election.cc:288] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:34:20.381860 4932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:20.382114 4932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [term 1 LEADER]: Becoming Leader. State: Replica: 9555cf68cd544f65a2f976b0981edb2b, State: Running, Role: LEADER
I20260925 04:34:20.382400 4932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } }
I20260925 04:34:20.382838 4927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:34:20.384017 4933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9555cf68cd544f65a2f976b0981edb2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } } }
I20260925 04:34:20.384008 4934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9555cf68cd544f65a2f976b0981edb2b. Latest consensus state: current_term: 1 leader_uuid: "9555cf68cd544f65a2f976b0981edb2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9555cf68cd544f65a2f976b0981edb2b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35387 } } }
I20260925 04:34:20.384178 4933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:20.384178 4934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:20.384681 4941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:34:20.387552 4941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:34:20.392397 4941 catalog_manager.cc:1347] Generated new cluster ID: 6c3df1ac3c03486ba0403118dbb4205c
I20260925 04:34:20.392457 4941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:34:20.403009 4941 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:34:20.403867 4941 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:34:20.409626 4941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9555cf68cd544f65a2f976b0981edb2b: Generated new TSK 0
I20260925 04:34:20.410236 4941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:34:20.416072 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--builtin_ntp_servers=127.27.46.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 04:34:20.529351 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:20.529680 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:20.529803 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:20.533972 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:20.534025 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:20.534118 4952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:34:20.538838 4952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:20.539953 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:20.541368 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:20.551204 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.551306 4952 server_base.cc:1034] running on GCE node
W20260925 04:34:20.551211 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:20.551211 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.551967 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:20.552662 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:20.553848 4952 hybrid_clock.cc:648] HybridClock initialized: now 1790310860553814 us; error 45 us; skew 500 ppm
I20260925 04:34:20.555660 4952 webserver.cc:460] Webserver started at http://127.27.46.193:36823/ using document root <none> and password file <none>
I20260925 04:34:20.556260 4952 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:20.556352 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:20.556586 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:20.558740 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "8663b5b423744497b07a9fdde88174cf"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "174cd8525ae1889515b35681742e159d"
server_key_iv: "3863e2a207797479aae5e96002b5811d"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.559315 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "8663b5b423744497b07a9fdde88174cf"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "174cd8525ae1889515b35681742e159d"
server_key_iv: "3863e2a207797479aae5e96002b5811d"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.562954 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:34:20.565546 4966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.566974 4952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:20.567070 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "8663b5b423744497b07a9fdde88174cf"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "174cd8525ae1889515b35681742e159d"
server_key_iv: "3863e2a207797479aae5e96002b5811d"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.567159 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:20.581727 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:20.582434 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:20.582597 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:20.583217 4952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:20.584340 4952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:20.584389 4952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.584431 4952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:20.584481 4952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.593334 4952 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:40559
I20260925 04:34:20.593360 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:40559 every 8 connection(s)
I20260925 04:34:20.594304 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:34:20.602802 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 4952
I20260925 04:34:20.602922 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:34:20.603263 27835 external_mini_cluster.cc:1428] Setting key 3d66f27870cba2bf3f997cab5e043fb7
I20260925 04:34:20.603813 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.596243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:47457 (local address 127.27.46.254:35387)
0925 04:34:20.596392 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:20.596395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:20.597517 (+ 1122us) server_negotiation.cc:407] Connection header received
0925 04:34:20.599086 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:20.599088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:20.599136 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:20.599211 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:20.600188 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.600718 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.601887 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.602100 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.602181 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.603321 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:20.603326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:20.603329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:20.603345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:20.603368 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:20.603380 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:20.603456 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:20.603613 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 04:34:20.603680 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:34:20.604796 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.596676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35387 (local address 127.27.46.193:47457)
0925 04:34:20.597329 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:20.597369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:20.598871 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:20.599474 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:20.599481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:20.599566 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:20.600056 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.600069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.600829 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.600831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.601785 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.601794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.602236 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.602239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.602375 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.603057 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:20.603059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:20.603101 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:20.603166 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:20.603458 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:20.603466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:20.603470 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:34:20.603580 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:34:20.603869 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260925 04:34:20.605405 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--builtin_ntp_servers=127.27.46.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:34:20.606835 5081 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35387
I20260925 04:34:20.607115 5081 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:20.607765 5081 heartbeater.cc:511] Master 127.27.46.254:35387 requested a full tablet report, sending...
I20260925 04:34:20.609371 4891 ts_manager.cc:194] Registered new tserver with Master: 8663b5b423744497b07a9fdde88174cf (127.27.46.193:40559)
I20260925 04:34:20.610576 4891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.193:47457
W20260925 04:34:20.723052 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:20.723412 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:20.723487 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:20.727752 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:20.727849 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:20.727944 5084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:34:20.733018 5084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:20.734275 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:20.735707 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:20.745486 5090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:20.745491 5089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.745666 5084 server_base.cc:1034] running on GCE node
W20260925 04:34:20.745608 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.746263 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:20.746917 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:20.748106 5084 hybrid_clock.cc:648] HybridClock initialized: now 1790310860748085 us; error 32 us; skew 500 ppm
I20260925 04:34:20.749956 5084 webserver.cc:460] Webserver started at http://127.27.46.194:33559/ using document root <none> and password file <none>
I20260925 04:34:20.750530 5084 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:20.750633 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:20.750870 5084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:20.752957 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "5bcd56de29204247b8eae9fe437a5ef4"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "9645b2c7db31f8b49ee4727c50ecab80"
server_key_iv: "88dff4b37707a1aebac76f6079ec7a96"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.753578 5084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "5bcd56de29204247b8eae9fe437a5ef4"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "9645b2c7db31f8b49ee4727c50ecab80"
server_key_iv: "88dff4b37707a1aebac76f6079ec7a96"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.757228 5084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:34:20.759799 5098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.761010 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:34:20.761150 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "5bcd56de29204247b8eae9fe437a5ef4"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "9645b2c7db31f8b49ee4727c50ecab80"
server_key_iv: "88dff4b37707a1aebac76f6079ec7a96"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.761276 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:20.788115 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:20.788906 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:20.789120 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:20.789819 5084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:20.790794 5084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:20.790866 5084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.790944 5084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:20.791005 5084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.800107 5084 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:45333
I20260925 04:34:20.800154 5211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:45333 every 8 connection(s)
I20260925 04:34:20.801074 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:34:20.802738 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5084
I20260925 04:34:20.802872 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:34:20.803190 27835 external_mini_cluster.cc:1428] Setting key bc6f98edf11bd29eb4ce58567ac681aa
I20260925 04:34:20.805238 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--builtin_ntp_servers=127.27.46.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:34:20.810823 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.802984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52553 (local address 127.27.46.254:35387)
0925 04:34:20.803128 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:20.803133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:20.804404 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 04:34:20.805929 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:20.805935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:20.806001 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:20.806096 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:20.807053 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.807580 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.808744 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.809051 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.809162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.810297 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:20.810301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:20.810304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:20.810318 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:20.810337 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:20.810352 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:20.810428 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:20.810612 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 04:34:20.810678 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:34:20.811729 5214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.803463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35387 (local address 127.27.46.194:52553)
0925 04:34:20.804231 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:20.804287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:20.805709 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:20.806270 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:20.806276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:20.806359 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:20.806892 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.806904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.807721 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.807724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.808610 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.808619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.809219 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.809223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.809388 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.810028 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:20.810029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:20.810073 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:20.810137 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:20.810442 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:20.810454 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:20.810462 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:34:20.810598 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 04:34:20.810880 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260925 04:34:20.812924 5212 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35387
I20260925 04:34:20.813201 5212 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:20.813925 5212 heartbeater.cc:511] Master 127.27.46.254:35387 requested a full tablet report, sending...
I20260925 04:34:20.815127 4891 ts_manager.cc:194] Registered new tserver with Master: 5bcd56de29204247b8eae9fe437a5ef4 (127.27.46.194:45333)
I20260925 04:34:20.815703 4891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.194:52553
W20260925 04:34:20.919883 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:20.920169 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:20.920208 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:20.924412 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:20.924482 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:20.924542 5215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:34:20.929605 5215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:20.930768 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:20.932123 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:34:20.941784 5215 server_base.cc:1034] running on GCE node
W20260925 04:34:20.941764 5224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:20.941983 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:20.942008 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:20.942492 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:20.943082 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:20.944247 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790310860944229 us; error 28 us; skew 500 ppm
I20260925 04:34:20.946170 5215 webserver.cc:460] Webserver started at http://127.27.46.195:46843/ using document root <none> and password file <none>
I20260925 04:34:20.946753 5215 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:20.946816 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:20.947080 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:20.949241 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "e7826553106149cb8f3d7c430ec5b56b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "a599808e9f8f0dd6c1d9ab8bd32819bd"
server_key_iv: "3f598f1094f6df7b63282c836246a5aa"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.949890 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "e7826553106149cb8f3d7c430ec5b56b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "a599808e9f8f0dd6c1d9ab8bd32819bd"
server_key_iv: "3f598f1094f6df7b63282c836246a5aa"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.953770 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:34:20.956401 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.957794 5215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:34:20.957904 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "e7826553106149cb8f3d7c430ec5b56b"
format_stamp: "Formatted at 2026-09-25 04:34:20 on dist-test-slave-r8r3"
server_key: "a599808e9f8f0dd6c1d9ab8bd32819bd"
server_key_iv: "3f598f1094f6df7b63282c836246a5aa"
server_key_version: "encryptionkey@0"
I20260925 04:34:20.957998 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:20.971824 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:20.972549 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:20.972708 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:20.973253 5215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:20.974208 5215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:20.974256 5215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.974299 5215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:20.974315 5215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:20.983135 5215 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:40289
I20260925 04:34:20.983265 5344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:40289 every 8 connection(s)
I20260925 04:34:20.984154 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:34:20.992866 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5215
I20260925 04:34:20.992997 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:34:20.993337 27835 external_mini_cluster.cc:1428] Setting key 8fb3aaa4b5a527fcebf381a1f9023397
I20260925 04:34:20.993618 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.986135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36163 (local address 127.27.46.254:35387)
0925 04:34:20.986310 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:20.986314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:20.987388 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 04:34:20.988768 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:20.988771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:20.988821 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:20.988905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:20.989819 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.990390 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.991527 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.991744 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.991829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.992986 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:20.992993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:20.992998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:20.993016 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:20.993045 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:20.993074 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:20.993153 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:20.993331 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 04:34:20.993419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:34:20.994580 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:20.986570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35387 (local address 127.27.46.195:36163)
0925 04:34:20.987227 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:20.987269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:20.988536 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:20.989080 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:20.989087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:20.989172 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:20.989679 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.989693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.990503 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.990506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.991390 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:20.991399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:20.991884 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:20.991886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:20.992028 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:20.992645 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:20.992648 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:20.992706 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:20.992793 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:20.993162 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:20.993173 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:20.993180 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:34:20.993321 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 04:34:20.993650 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260925 04:34:20.995740 5345 heartbeater.cc:347] Connected to a master server at 127.27.46.254:35387
I20260925 04:34:20.995985 5345 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:20.996510 5345 heartbeater.cc:511] Master 127.27.46.254:35387 requested a full tablet report, sending...
I20260925 04:34:20.997884 4891 ts_manager.cc:194] Registered new tserver with Master: e7826553106149cb8f3d7c430ec5b56b (127.27.46.195:40289)
I20260925 04:34:20.998451 4891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.195:36163
I20260925 04:34:21.007354 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:34:21.010329 4930 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:21.009353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.27.46.254:35387)
0925 04:34:21.009540 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.009558 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.009719 (+ 161us) server_negotiation.cc:407] Connection header received
0925 04:34:21.009826 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.009828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.009872 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.009943 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.010164 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34818: BlockingRecv error: recv got EOF from 127.0.0.1:34818 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:34:21.010560 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4952
I20260925 04:34:21.017275 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5084
I20260925 04:34:21.025408 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5215
I20260925 04:34:21.031600 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 4859
2026-09-25T04:34:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (881 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T04:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:21Z Disabled control of system clock
I20260925 04:34:21.061110 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:32815
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:34:21.170881 5356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:21.171152 5356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:21.171190 5356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:21.175173 5356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:21.175230 5356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:21.175244 5356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:21.175261 5356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:21.175276 5356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:21.179962 5356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:32815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:21.181031 5356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:21.182494 5356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:21.190960 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.191080 5356 server_base.cc:1034] running on GCE node
W20260925 04:34:21.190960 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.191023 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.191908 5356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:21.193135 5356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:21.194325 5356 hybrid_clock.cc:648] HybridClock initialized: now 1790310861194310 us; error 28 us; skew 500 ppm
I20260925 04:34:21.196210 5356 webserver.cc:460] Webserver started at http://127.27.46.254:41719/ using document root <none> and password file <none>
I20260925 04:34:21.196812 5356 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:21.196907 5356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:21.197149 5356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:21.199484 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "17eb7a7d47944ba887100676c1f0af8a"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "0581a272ecc0a18f95c967a4a5feafd7"
server_key_iv: "2e2b5116c71581fbca1152fe391effef"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.200074 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "17eb7a7d47944ba887100676c1f0af8a"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "0581a272ecc0a18f95c967a4a5feafd7"
server_key_iv: "2e2b5116c71581fbca1152fe391effef"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.203930 5356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:34:21.206399 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.207615 5356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:21.207755 5356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "17eb7a7d47944ba887100676c1f0af8a"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "0581a272ecc0a18f95c967a4a5feafd7"
server_key_iv: "2e2b5116c71581fbca1152fe391effef"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.207871 5356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:21.212491 5356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:21.213099 5356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:21.213284 5356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:21.220059 5356 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:32815
I20260925 04:34:21.220086 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:32815 every 8 connection(s)
I20260925 04:34:21.221012 5356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:34:21.224262 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:21.226858 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5356
I20260925 04:34:21.226984 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:34:21.227275 27835 external_mini_cluster.cc:1428] Setting key 2fab8858c6ea8ba5bfe34d8e8fd485fd
I20260925 04:34:21.230670 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a: Bootstrap starting.
I20260925 04:34:21.232982 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.228791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.27.46.254:32815)
0925 04:34:21.229437 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.229452 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.229486 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:34:21.230187 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.230203 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.230554 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.230893 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.231682 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:21.231687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:21.231698 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:21.231708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:21.231763 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:21.231780 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:21.231888 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:21.232029 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 04:34:21.232305 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260925 04:34:21.233275 5425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:21.234139 5425 log.cc:824] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:21.236105 5425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a: No bootstrap required, opened a new log
I20260925 04:34:21.239529 5425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } }
I20260925 04:34:21.239744 5425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:21.239795 5425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17eb7a7d47944ba887100676c1f0af8a, State: Initialized, Role: FOLLOWER
I20260925 04:34:21.240149 5425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } }
I20260925 04:34:21.240250 5425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:34:21.240312 5425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:34:21.240381 5425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:21.241182 5425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } }
I20260925 04:34:21.241468 5425 leader_election.cc:302] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17eb7a7d47944ba887100676c1f0af8a; no voters:
I20260925 04:34:21.241791 5425 leader_election.cc:288] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:34:21.241909 5430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:21.242189 5430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [term 1 LEADER]: Becoming Leader. State: Replica: 17eb7a7d47944ba887100676c1f0af8a, State: Running, Role: LEADER
I20260925 04:34:21.242581 5430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } }
I20260925 04:34:21.243148 5425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:34:21.244369 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17eb7a7d47944ba887100676c1f0af8a. Latest consensus state: current_term: 1 leader_uuid: "17eb7a7d47944ba887100676c1f0af8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } } }
I20260925 04:34:21.244504 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:21.244705 5431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17eb7a7d47944ba887100676c1f0af8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17eb7a7d47944ba887100676c1f0af8a" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 32815 } } }
I20260925 04:34:21.244874 5431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:21.244917 5440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:34:21.248068 5440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:34:21.254118 5440 catalog_manager.cc:1347] Generated new cluster ID: e8654c670cc84a779c24439e5406729d
I20260925 04:34:21.254204 5440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:34:21.265973 5440 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:34:21.266276 5440 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:34:21.271482 5440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17eb7a7d47944ba887100676c1f0af8a: Generated new TSK 0
I20260925 04:34:21.272222 5440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:34:21.287439 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--builtin_ntp_servers=127.27.46.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:34:21.420746 5450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:21.421056 5450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:21.421096 5450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:21.425283 5450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:21.425349 5450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:21.425408 5450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:34:21.430295 5450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:21.431378 5450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:21.432765 5450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:21.442387 5456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.442435 5455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.442580 5458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.442629 5450 server_base.cc:1034] running on GCE node
I20260925 04:34:21.443244 5450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:21.444026 5450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:21.445222 5450 hybrid_clock.cc:648] HybridClock initialized: now 1790310861445186 us; error 48 us; skew 500 ppm
I20260925 04:34:21.447232 5450 webserver.cc:460] Webserver started at http://127.27.46.193:35573/ using document root <none> and password file <none>
I20260925 04:34:21.447922 5450 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:21.448027 5450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:21.448278 5450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:21.450608 5450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "8fd5b146fb0c42198953d610d2cae8b6"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "9b4e565987e1c01db935749d2c162cb0"
server_key_iv: "33dcff8c995b7c92df830e62e6b5f794"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.451233 5450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd5b146fb0c42198953d610d2cae8b6"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "9b4e565987e1c01db935749d2c162cb0"
server_key_iv: "33dcff8c995b7c92df830e62e6b5f794"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.455374 5450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:34:21.458178 5464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.459445 5450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:21.459600 5450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "8fd5b146fb0c42198953d610d2cae8b6"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "9b4e565987e1c01db935749d2c162cb0"
server_key_iv: "33dcff8c995b7c92df830e62e6b5f794"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.459731 5450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:21.464891 5450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:21.465620 5450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:21.465817 5450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:21.466403 5450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:21.467381 5450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:21.467458 5450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.467535 5450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:21.467597 5450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.477384 5450 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:40501
I20260925 04:34:21.477398 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:40501 every 8 connection(s)
I20260925 04:34:21.478400 5450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:34:21.484516 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.480296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48437 (local address 127.27.46.254:32815)
0925 04:34:21.480442 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.480446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.481579 (+ 1133us) server_negotiation.cc:407] Connection header received
0925 04:34:21.482839 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.482842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.482888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.482969 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.484083 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:21.484086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:21.484088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:21.484100 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:21.484119 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:21.484128 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:21.484211 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:21.484366 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 04:34:21.484410 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:34:21.485013 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5450
I20260925 04:34:21.485148 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:34:21.485431 27835 external_mini_cluster.cc:1428] Setting key b1647c73adcbea37931f5eb7063c069a
I20260925 04:34:21.485427 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.480724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:32815 (local address 127.27.46.193:48437)
0925 04:34:21.481378 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:21.481417 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:21.482653 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:21.483124 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:21.483131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:21.483214 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:21.483824 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:21.483826 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:21.483866 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:21.483934 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:21.484208 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:21.484214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:21.484218 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:34:21.484349 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:34:21.484544 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260925 04:34:21.486856 5579 heartbeater.cc:347] Connected to a master server at 127.27.46.254:32815
I20260925 04:34:21.487118 5579 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:21.487545 5579 heartbeater.cc:511] Master 127.27.46.254:32815 requested a full tablet report, sending...
I20260925 04:34:21.487865 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--builtin_ntp_servers=127.27.46.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 04:34:21.489583 5388 ts_manager.cc:194] Registered new tserver with Master: 8fd5b146fb0c42198953d610d2cae8b6 (127.27.46.193:40501)
W20260925 04:34:21.623152 5582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:21.625195 5582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:21.625316 5582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:21.630656 5582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:21.630751 5582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:21.630818 5582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:34:21.637316 5582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:21.638825 5582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:21.640575 5582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:21.655869 5590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.655988 5587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.656126 5588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.657374 5582 server_base.cc:1034] running on GCE node
I20260925 04:34:21.658099 5582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:21.659649 5582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:21.660924 5582 hybrid_clock.cc:648] HybridClock initialized: now 1790310861660975 us; error 99 us; skew 500 ppm
I20260925 04:34:21.664718 5582 webserver.cc:460] Webserver started at http://127.27.46.194:46271/ using document root <none> and password file <none>
I20260925 04:34:21.665546 5582 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:21.665668 5582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:21.665939 5582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:21.668480 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "242fd65c9c7745f7a424756240e24b67"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "ed400765688ebeabe6782fb08230abb1"
server_key_iv: "5ac946a6e9291dc2dca9eb3c3a4c3165"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.670933 5582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "242fd65c9c7745f7a424756240e24b67"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "ed400765688ebeabe6782fb08230abb1"
server_key_iv: "5ac946a6e9291dc2dca9eb3c3a4c3165"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.676185 5582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:34:21.680770 5596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.682436 5582 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 04:34:21.682628 5582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "242fd65c9c7745f7a424756240e24b67"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "ed400765688ebeabe6782fb08230abb1"
server_key_iv: "5ac946a6e9291dc2dca9eb3c3a4c3165"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.682793 5582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:21.690657 5582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:21.691493 5582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:21.691926 5582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:21.693171 5582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:21.695919 5582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:21.696038 5582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.696121 5582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:21.696336 5582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.709188 5582 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:45613
I20260925 04:34:21.709201 5710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:45613 every 8 connection(s)
I20260925 04:34:21.710523 5582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:34:21.717634 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5582
I20260925 04:34:21.718165 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:34:21.718592 27835 external_mini_cluster.cc:1428] Setting key c76a2d4f42a49481cc52059aa81a819b
I20260925 04:34:21.719937 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.713431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:47291 (local address 127.27.46.254:32815)
0925 04:34:21.713673 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.713677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.714937 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 04:34:21.716365 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.716368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.716414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.716514 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.719015 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:21.719049 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:21.719055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:21.719081 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:21.719112 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:21.719124 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:21.719252 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:21.719738 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 04:34:21.719795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260925 04:34:21.721081 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.713937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:32815 (local address 127.27.46.194:47291)
0925 04:34:21.714743 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:21.714798 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:21.716089 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:21.716910 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:21.716919 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:21.717312 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:21.718503 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:21.718505 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:21.718572 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:21.718694 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:21.719390 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:21.719404 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:21.719411 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:34:21.719647 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 04:34:21.719897 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":131,"threads_started":1}
I20260925 04:34:21.721342 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--builtin_ntp_servers=127.27.46.212:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 04:34:21.723536 5711 heartbeater.cc:347] Connected to a master server at 127.27.46.254:32815
I20260925 04:34:21.723842 5711 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:21.724319 5711 heartbeater.cc:511] Master 127.27.46.254:32815 requested a full tablet report, sending...
I20260925 04:34:21.725903 5388 ts_manager.cc:194] Registered new tserver with Master: 242fd65c9c7745f7a424756240e24b67 (127.27.46.194:45613)
W20260925 04:34:21.845715 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:21.846009 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:21.846096 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:21.850106 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:21.850180 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:21.850274 5714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:34:21.855218 5714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:21.856297 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:21.857683 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:21.867362 5720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:21.867347 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.867470 5714 server_base.cc:1034] running on GCE node
W20260925 04:34:21.867348 5723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:21.868080 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:21.868682 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:21.869854 5714 hybrid_clock.cc:648] HybridClock initialized: now 1790310861869838 us; error 29 us; skew 500 ppm
I20260925 04:34:21.871615 5714 webserver.cc:460] Webserver started at http://127.27.46.195:39025/ using document root <none> and password file <none>
I20260925 04:34:21.872171 5714 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:21.872231 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:21.872395 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:21.874495 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "22ca1a5c122a4d7699e084aa33c012a8"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "6c53a7eec796aa485559f515251b66a8"
server_key_iv: "ff39c33c0a649c79d8af95ed9f91a1dc"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.875001 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "22ca1a5c122a4d7699e084aa33c012a8"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "6c53a7eec796aa485559f515251b66a8"
server_key_iv: "ff39c33c0a649c79d8af95ed9f91a1dc"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.878731 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:34:21.881268 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.882645 5714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:21.882763 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "22ca1a5c122a4d7699e084aa33c012a8"
format_stamp: "Formatted at 2026-09-25 04:34:21 on dist-test-slave-r8r3"
server_key: "6c53a7eec796aa485559f515251b66a8"
server_key_iv: "ff39c33c0a649c79d8af95ed9f91a1dc"
server_key_version: "encryptionkey@0"
I20260925 04:34:21.882872 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:21.887624 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:21.888310 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:21.888537 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:21.889132 5714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:21.890362 5714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:21.890414 5714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.890453 5714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:21.890468 5714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:21.898973 5714 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:40019
I20260925 04:34:21.899004 5842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:40019 every 8 connection(s)
I20260925 04:34:21.899874 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:34:21.906163 5428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.901755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52411 (local address 127.27.46.254:32815)
0925 04:34:21.901901 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.901905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.903089 (+ 1184us) server_negotiation.cc:407] Connection header received
0925 04:34:21.904408 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.904412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.904460 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.904565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.905675 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:21.905678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:21.905681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:34:21.905692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:21.905713 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:34:21.905724 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:21.905823 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:21.905970 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 04:34:21.906017 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:34:21.906965 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:21.902236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:32815 (local address 127.27.46.195:52411)
0925 04:34:21.902896 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:21.902947 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:21.904194 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:21.904736 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:21.904744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:21.904842 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:21.905360 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:34:21.905361 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:21.905399 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:34:21.905466 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:21.905816 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:21.905824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:21.905832 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:34:21.906013 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:34:21.906246 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260925 04:34:21.908057 5843 heartbeater.cc:347] Connected to a master server at 127.27.46.254:32815
I20260925 04:34:21.908277 5843 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:21.908694 5843 heartbeater.cc:511] Master 127.27.46.254:32815 requested a full tablet report, sending...
I20260925 04:34:21.908870 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5714
I20260925 04:34:21.908981 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:34:21.909271 27835 external_mini_cluster.cc:1428] Setting key 46798dc4edbc80627f73df3f0f314c82
I20260925 04:34:21.909869 5388 ts_manager.cc:194] Registered new tserver with Master: 22ca1a5c122a4d7699e084aa33c012a8 (127.27.46.195:40019)
I20260925 04:34:21.911311 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:34:21.914137 5428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:34:21.913336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.27.46.254:32815)
0925 04:34:21.913460 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:21.913463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:21.913671 (+ 208us) server_negotiation.cc:407] Connection header received
0925 04:34:21.913769 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:21.913771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:21.913799 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:21.913862 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:21.914052 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36560: BlockingRecv error: recv got EOF from 127.0.0.1:36560 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:34:21.914428 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5450
I20260925 04:34:21.920905 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5582
I20260925 04:34:21.926648 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5714
I20260925 04:34:21.932340 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5356
2026-09-25T04:34:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (900 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:34:21 dist-test-slave-r8r3 krb5kdc[5853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:34:21 dist-test-slave-r8r3 krb5kdc[5853](info): set up 2 sockets
Sep 25 04:34:21 dist-test-slave-r8r3 krb5kdc[5853](info): commencing operation
krb5kdc: starting...
W20260925 04:34:22.978569 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:34:23.102773 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45745
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:23.214851 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:23.215129 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:23.215171 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:23.219317 5867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:34:23.219374 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:23.219394 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:23.219411 5867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:34:23.219449 5867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:34:23.224779 5867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:45745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:23.226009 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:23.227069 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:23.236064 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.236104 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.236133 5867 server_base.cc:1034] running on GCE node
W20260925 04:34:23.236071 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.236899 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:23.238094 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:23.239292 5867 hybrid_clock.cc:648] HybridClock initialized: now 1790310863239272 us; error 32 us; skew 500 ppm
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:23.247473 5867 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:34:23.248368 5867 webserver.cc:460] Webserver started at http://127.27.46.254:40223/ using document root <none> and password file <none>
I20260925 04:34:23.248965 5867 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:23.249028 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:23.249207 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:23.251390 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "32793012d70e4928bd07ff3fb014c2f4"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "94f3400256f4de9e984bbed34a5f4fcf"
server_key_iv: "805b36d7582996b9fada16ed681207cf"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.251909 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "32793012d70e4928bd07ff3fb014c2f4"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "94f3400256f4de9e984bbed34a5f4fcf"
server_key_iv: "805b36d7582996b9fada16ed681207cf"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.255628 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:34:23.258095 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.259418 5867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:23.259518 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "32793012d70e4928bd07ff3fb014c2f4"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "94f3400256f4de9e984bbed34a5f4fcf"
server_key_iv: "805b36d7582996b9fada16ed681207cf"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.259606 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:23.272727 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:23.275571 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:23.275724 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:23.283324 5867 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:45745
I20260925 04:34:23.283420 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:45745 every 8 connection(s)
I20260925 04:34:23.284351 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:34:23.287209 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:23.289592 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5867
I20260925 04:34:23.289731 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:34:23.290050 27835 external_mini_cluster.cc:1428] Setting key bed96a287cdef4b4b26194f9607565e5
I20260925 04:34:23.293797 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: Bootstrap starting.
I20260925 04:34:23.296485 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:23.297365 5935 log.cc:824] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:23.299764 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: No bootstrap required, opened a new log
I20260925 04:34:23.304581 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } }
I20260925 04:34:23.304857 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:23.304930 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32793012d70e4928bd07ff3fb014c2f4, State: Initialized, Role: FOLLOWER
I20260925 04:34:23.305462 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } }
I20260925 04:34:23.305648 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:34:23.305651 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:23.291546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.27.46.254:45745)
0925 04:34:23.291976 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:23.291992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:23.292031 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:34:23.292708 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:23.292724 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:23.293061 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:23.293345 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:23.294372 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.295342 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.296611 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.297092 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.297212 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:23.300310 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:23.300317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:23.300332 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:23.300374 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:23.302414 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.302830 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.302836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.302841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.302986 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.303249 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.303252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.303254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.303785 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:23.304002 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:23.304182 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 04:34:23.304488 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260925 04:34:23.305735 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:34:23.305821 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:23.306938 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } }
I20260925 04:34:23.307277 5935 leader_election.cc:302] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32793012d70e4928bd07ff3fb014c2f4; no voters:
I20260925 04:34:23.307585 5935 leader_election.cc:288] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:34:23.307699 5940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:23.307916 5940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [term 1 LEADER]: Becoming Leader. State: Replica: 32793012d70e4928bd07ff3fb014c2f4, State: Running, Role: LEADER
I20260925 04:34:23.308225 5940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } }
I20260925 04:34:23.308831 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:34:23.309860 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32793012d70e4928bd07ff3fb014c2f4. Latest consensus state: current_term: 1 leader_uuid: "32793012d70e4928bd07ff3fb014c2f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } } }
I20260925 04:34:23.309980 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:23.310385 5949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:34:23.310310 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32793012d70e4928bd07ff3fb014c2f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32793012d70e4928bd07ff3fb014c2f4" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45745 } } }
I20260925 04:34:23.311208 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:34:23.313758 5949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:34:23.319940 5949 catalog_manager.cc:1347] Generated new cluster ID: d44695daa4ed4c54981b6b558cc9302b
I20260925 04:34:23.320025 5949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:34:23.328763 5949 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:34:23.330022 5949 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:34:23.341408 5949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: Generated new TSK 0
I20260925 04:34:23.342335 5949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:34:23.377089 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:23.493490 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:23.493757 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:23.493798 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:23.497695 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:23.497741 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:23.497794 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:34:23.503265 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:23.504328 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:23.505440 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:23.515422 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.515466 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.515422 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.515658 5961 server_base.cc:1034] running on GCE node
I20260925 04:34:23.516137 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:23.516841 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:23.518028 5961 hybrid_clock.cc:648] HybridClock initialized: now 1790310863518015 us; error 28 us; skew 500 ppm
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:23.520843 5961 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:34:23.521785 5961 webserver.cc:460] Webserver started at http://127.27.46.193:33001/ using document root <none> and password file <none>
I20260925 04:34:23.522361 5961 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:23.522451 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:23.522693 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:23.524755 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "08c9eafcaea347be85ac054981ab0607"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c401b655a1f4da521a76de4d4547f95a"
server_key_iv: "e192e55e5f43af7329ef1fed4d72ada8"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.525334 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "08c9eafcaea347be85ac054981ab0607"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c401b655a1f4da521a76de4d4547f95a"
server_key_iv: "e192e55e5f43af7329ef1fed4d72ada8"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.529610 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:34:23.532188 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.533461 5961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:34:23.533653 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "08c9eafcaea347be85ac054981ab0607"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c401b655a1f4da521a76de4d4547f95a"
server_key_iv: "e192e55e5f43af7329ef1fed4d72ada8"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.533774 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:23.556890 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:23.560362 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:23.560576 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:23.561221 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:23.562131 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:23.562212 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.562306 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:23.562361 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.571915 5961 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44001
I20260925 04:34:23.571943 6090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44001 every 8 connection(s)
I20260925 04:34:23.572829 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:34:23.573802 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 5961
I20260925 04:34:23.573920 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:34:23.574229 27835 external_mini_cluster.cc:1428] Setting key ee2b9c7f8bdef078305cf4676f6dd370
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:34:23.587903 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:23.574882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:38219 (local address 127.27.46.254:45745)
0925 04:34:23.575047 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:23.575052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:23.575997 (+ 945us) server_negotiation.cc:407] Connection header received
0925 04:34:23.577468 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:23.577474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:23.577571 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:23.577665 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:23.578824 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.579726 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.580930 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.581367 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.581533 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:23.584004 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:23.584008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:23.584011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:23.584052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:23.585580 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.586118 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.586121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.586122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.586200 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.586584 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.586588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.586590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.586848 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:23.586962 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:23.587483 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 04:34:23.587636 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:34:23.588805 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:23.575361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45745 (local address 127.27.46.193:38219)
0925 04:34:23.575827 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:23.575880 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:23.577275 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:23.577826 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:23.577832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:23.578191 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:23.578663 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:23.578677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.579890 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.579893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:23.580764 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:23.580781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.581574 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.581578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:23.581720 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:23.582329 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:23.582346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:23.583831 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:23.585698 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:23.585703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:23.585715 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:23.586024 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:23.586292 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:23.586295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:23.586297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:23.586470 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:23.586972 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:23.586978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:23.587232 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 04:34:23.587400 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:34:23.587667 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
I20260925 04:34:23.590227 6091 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45745
I20260925 04:34:23.590502 6091 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:23.591158 6091 heartbeater.cc:511] Master 127.27.46.254:45745 requested a full tablet report, sending...
I20260925 04:34:23.593003 5899 ts_manager.cc:194] Registered new tserver with Master: 08c9eafcaea347be85ac054981ab0607 (127.27.46.193:44001)
I20260925 04:34:23.594704 5899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:38219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:34:23.604571 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:23.717378 6096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:23.717649 6096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:23.717691 6096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:23.721639 6096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:23.721690 6096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:23.721747 6096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:34:23.726990 6096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:23.728011 6096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:23.729019 6096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:23.738426 6104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.738448 6101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.738426 6102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.738898 6096 server_base.cc:1034] running on GCE node
I20260925 04:34:23.739287 6096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:23.739934 6096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:23.741096 6096 hybrid_clock.cc:648] HybridClock initialized: now 1790310863741068 us; error 39 us; skew 500 ppm
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:23.743806 6096 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:34:23.744668 6096 webserver.cc:460] Webserver started at http://127.27.46.194:39537/ using document root <none> and password file <none>
I20260925 04:34:23.745229 6096 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:23.745286 6096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:23.745585 6096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:23.747649 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "ce17078744b644fb94db256077e82420"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c1a62d475d06da2e0b603e755e60c5af"
server_key_iv: "c665a38f61dad89fb1e3d214d33addc1"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.748236 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "ce17078744b644fb94db256077e82420"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c1a62d475d06da2e0b603e755e60c5af"
server_key_iv: "c665a38f61dad89fb1e3d214d33addc1"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.751957 6096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 04:34:23.754392 6111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.755621 6096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:34:23.755744 6096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "ce17078744b644fb94db256077e82420"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "c1a62d475d06da2e0b603e755e60c5af"
server_key_iv: "c665a38f61dad89fb1e3d214d33addc1"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.755844 6096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:23.779148 6096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:23.782115 6096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:23.782320 6096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:23.782923 6096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:23.783763 6096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:23.783816 6096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.783913 6096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:23.783955 6096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.793462 6096 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:46045
I20260925 04:34:23.793488 6228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:46045 every 8 connection(s)
I20260925 04:34:23.794390 6096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:34:23.801272 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 6096
I20260925 04:34:23.801405 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:34:23.801731 27835 external_mini_cluster.cc:1428] Setting key eb8c076d772cf004214a145f744aef85
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:23.809794 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:23.796214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:44569 (local address 127.27.46.254:45745)
0925 04:34:23.796371 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:23.796375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:23.797233 (+ 858us) server_negotiation.cc:407] Connection header received
0925 04:34:23.798685 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:23.798688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:23.798736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:23.798812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:23.799978 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.800476 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.802027 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.802298 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.802424 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:23.805660 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:23.805663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:23.805665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:23.805689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:23.807217 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.807861 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.807867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.807872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.807967 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:23.808342 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:23.808348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:23.808353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:23.808646 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:23.808761 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:23.809458 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 04:34:23.809622 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:34:23.810706 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:23.796575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45745 (local address 127.27.46.194:44569)
0925 04:34:23.797076 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:23.797117 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:23.798483 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:23.798979 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:23.798985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:23.799334 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:23.799830 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:23.799844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.800636 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.800640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:23.801834 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:23.801851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:23.802480 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:23.802484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:23.802711 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:23.803472 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:23.803499 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:23.805442 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:23.807408 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:23.807414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:23.807427 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:23.807721 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:23.808068 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:23.808071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:23.808072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:23.808226 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:23.808838 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:23.808847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:23.809161 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 04:34:23.809386 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 04:34:23.809753 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":139,"threads_started":1}
I20260925 04:34:23.812044 6229 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45745
I20260925 04:34:23.812307 6229 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:23.812798 6229 heartbeater.cc:511] Master 127.27.46.254:45745 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:34:23.813903 5899 ts_manager.cc:194] Registered new tserver with Master: ce17078744b644fb94db256077e82420 (127.27.46.194:46045)
I20260925 04:34:23.814509 5899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:44569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:34:23.832461 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:34:23.948375 6234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:34:23.948642 6234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:34:23.948698 6234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:34:23.952656 6234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:34:23.952720 6234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:34:23.952781 6234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:34:23.958110 6234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:34:23.959103 6234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:34:23.960201 6234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:34:23.970301 6239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.970431 6234 server_base.cc:1034] running on GCE node
W20260925 04:34:23.970296 6242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:34:23.970296 6240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:34:23.971083 6234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:34:23.971732 6234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:34:23.972950 6234 hybrid_clock.cc:648] HybridClock initialized: now 1790310863972935 us; error 26 us; skew 500 ppm
Sep 25 04:34:23 dist-test-slave-r8r3 krb5kdc[5853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:34:23.975971 6234 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:34:23.976805 6234 webserver.cc:460] Webserver started at http://127.27.46.195:39257/ using document root <none> and password file <none>
I20260925 04:34:23.977335 6234 fs_manager.cc:359] Metadata directory not provided
I20260925 04:34:23.977399 6234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:34:23.977648 6234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:34:23.979655 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "dbd23807d56d49518b6e36485e150735"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "1a6501d18c2313ea9409c244468dabe4"
server_key_iv: "11d74949dd64ef2a6a4a4b20df96e67e"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.980216 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "dbd23807d56d49518b6e36485e150735"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "1a6501d18c2313ea9409c244468dabe4"
server_key_iv: "11d74949dd64ef2a6a4a4b20df96e67e"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.984076 6234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:34:23.986574 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:23.987787 6234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:34:23.987920 6234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "dbd23807d56d49518b6e36485e150735"
format_stamp: "Formatted at 2026-09-25 04:34:23 on dist-test-slave-r8r3"
server_key: "1a6501d18c2313ea9409c244468dabe4"
server_key_iv: "11d74949dd64ef2a6a4a4b20df96e67e"
server_key_version: "encryptionkey@0"
I20260925 04:34:23.988008 6234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:34:24.013010 6234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:34:24.016484 6234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:34:24.016655 6234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:34:24.017257 6234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:34:24.018107 6234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:34:24.018160 6234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:24.018203 6234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:34:24.018229 6234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:34:24.027799 6234 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:37653
I20260925 04:34:24.027822 6364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:37653 every 8 connection(s)
I20260925 04:34:24.028795 6234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:34:24.029102 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 6234
I20260925 04:34:24.029234 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:34:24.029593 27835 external_mini_cluster.cc:1428] Setting key 304f2bfba60939c0be23e86e6ca781ce
Sep 25 04:34:24 dist-test-slave-r8r3 krb5kdc[5853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:34:24.042970 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.030698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:60189 (local address 127.27.46.254:45745)
0925 04:34:24.030814 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.030818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.032110 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 04:34:24.033422 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.033425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.033468 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.033611 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:24.034760 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.035289 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.036461 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.036755 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.036867 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.039271 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:24.039274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:24.039275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:24.039299 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:24.040732 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:24.041291 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:24.041294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:24.041295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:24.041375 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:24.041745 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:24.041748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:24.041749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:24.041924 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:24.042001 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.042677 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.042805 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:34:24.043855 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.031272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45745 (local address 127.27.46.195:60189)
0925 04:34:24.031952 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.032004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.033247 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.033798 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.033805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.034154 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:34:24.034619 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.034631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.035402 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.035404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.036314 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.036324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.036893 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.036896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.037033 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.037630 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:34:24.037681 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:34:24.039127 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:34:24.040829 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:24.040833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:24.040844 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:24.041081 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:24.041465 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:34:24.041467 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:34:24.041468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:34:24.041656 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:34:24.042070 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:34:24.042076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:34:24.042376 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.042622 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.042936 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":163,"threads_started":1}
I20260925 04:34:24.044893 6365 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45745
I20260925 04:34:24.045097 6365 heartbeater.cc:464] Registering TS with master...
I20260925 04:34:24.045567 6365 heartbeater.cc:511] Master 127.27.46.254:45745 requested a full tablet report, sending...
I20260925 04:34:24.046528 5899 ts_manager.cc:194] Registered new tserver with Master: dbd23807d56d49518b6e36485e150735 (127.27.46.195:37653)
I20260925 04:34:24.047049 5899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:60189
I20260925 04:34:24.055403 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:34:24.064738 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.057262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.27.46.254:45745)
0925 04:34:24.057383 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.057387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.057647 (+ 260us) server_negotiation.cc:407] Connection header received
0925 04:34:24.057744 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.057746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.057783 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.057847 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:34:24.058512 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.059049 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.059856 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.060079 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.060156 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.061377 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:34:24.061380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:34:24.061382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:34:24.061404 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:34:24.062927 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:24.063319 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:24.063323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:24.063326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:24.063410 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:34:24.063718 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:34:24.063722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:34:24.063724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:34:24.063943 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:34:24.064138 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.064441 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.064558 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:34:24.067698 5899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:34:24.070703 5899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:34:24.085079 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.078892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:46045 (local address 127.0.0.1:47344)
0925 04:34:24.079722 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.079773 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.079908 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.080728 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.080731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.080763 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:24.080967 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.080973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.082065 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.082068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.083469 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.083478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.084538 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.084542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.084661 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.084702 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.084810 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.084918 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":40,"spinlock_wait_cycles":7040,"thread_start_us":75,"threads_started":1}
I20260925 04:34:24.085433 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.079006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.27.46.194:46045)
0925 04:34:24.080257 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.080267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.080285 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:34:24.080366 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.080373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.080578 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.080757 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:24.081093 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.081931 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.083609 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.084406 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.084521 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.084670 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.085166 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.085279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":116,"threads_started":1}
I20260925 04:34:24.085811 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.079485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.27.46.195:37653)
0925 04:34:24.081065 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.081078 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.081097 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:34:24.081169 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.081172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.081294 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.081436 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:24.081696 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.082417 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.083846 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.084668 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.084774 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.084907 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.084995 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.085085 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":135,"threads_started":1}
I20260925 04:34:24.086043 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.079685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.27.46.193:44001)
0925 04:34:24.080218 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.080235 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.080259 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:34:24.080336 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.080343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.080559 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.080737 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:24.082130 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.083244 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.084388 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.085215 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.085318 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.085449 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.085620 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.085711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":87,"threads_started":1}
I20260925 04:34:24.085752 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.078960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44001 (local address 127.0.0.1:40076)
0925 04:34:24.079722 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.079773 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.079908 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.080725 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.080731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.080752 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:24.080956 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.080965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.083387 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.083390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.084267 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.084276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.085328 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.085332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.085415 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.085435 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.085549 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.085624 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":28,"spinlock_wait_cycles":4224,"thread_start_us":115,"threads_started":1}
I20260925 04:34:24.086447 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.079240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:37653 (local address 127.0.0.1:53258)
0925 04:34:24.079881 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.079897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.079997 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.081429 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.081432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.081450 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:24.081571 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.081581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.082552 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.082559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.083710 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.083722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.084777 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.084780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.084863 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.084885 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.084964 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.085034 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":78,"threads_started":1}
I20260925 04:34:24.088642 6024 tablet_service.cc:1467] Processing CreateTablet for tablet 067b58054353478c9f0b0a48c70461b3 (DEFAULT_TABLE table=test-table [id=f07f4b9245484a779f36cd09b404e80c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:24.088544 6162 tablet_service.cc:1467] Processing CreateTablet for tablet 067b58054353478c9f0b0a48c70461b3 (DEFAULT_TABLE table=test-table [id=f07f4b9245484a779f36cd09b404e80c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:24.089061 6299 tablet_service.cc:1467] Processing CreateTablet for tablet 067b58054353478c9f0b0a48c70461b3 (DEFAULT_TABLE table=test-table [id=f07f4b9245484a779f36cd09b404e80c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:34:24.089686 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067b58054353478c9f0b0a48c70461b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:24.089689 6162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067b58054353478c9f0b0a48c70461b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:24.089943 6299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067b58054353478c9f0b0a48c70461b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:34:24.095777 6383 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: Bootstrap starting.
I20260925 04:34:24.097265 6385 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Bootstrap starting.
I20260925 04:34:24.097807 6384 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Bootstrap starting.
I20260925 04:34:24.098611 6383 tablet_bootstrap.cc:654] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:24.099328 6383 log.cc:824] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:24.099771 6385 tablet_bootstrap.cc:654] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:24.099833 6384 tablet_bootstrap.cc:654] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Neither blocks nor log segments found. Creating new log.
I20260925 04:34:24.100649 6385 log.cc:824] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:24.100981 6384 log.cc:824] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Log is configured to *not* fsync() on all Append() calls
I20260925 04:34:24.101161 6383 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: No bootstrap required, opened a new log
I20260925 04:34:24.101361 6383 ts_tablet_manager.cc:1397] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 04:34:24.102528 6385 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: No bootstrap required, opened a new log
I20260925 04:34:24.102726 6385 ts_tablet_manager.cc:1397] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:34:24.104786 6383 raft_consensus.cc:348] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.104998 6383 raft_consensus.cc:374] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:24.105031 6383 raft_consensus.cc:729] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c9eafcaea347be85ac054981ab0607, State: Initialized, Role: FOLLOWER
I20260925 04:34:24.105531 6383 consensus_queue.cc:260] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.105763 6385 raft_consensus.cc:348] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.105999 6385 raft_consensus.cc:374] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:24.106062 6385 raft_consensus.cc:729] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce17078744b644fb94db256077e82420, State: Initialized, Role: FOLLOWER
I20260925 04:34:24.106307 6091 heartbeater.cc:503] Master 127.27.46.254:45745 was elected leader, sending a full tablet report...
I20260925 04:34:24.106459 6385 consensus_queue.cc:260] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.106740 6383 ts_tablet_manager.cc:1428] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 04:34:24.106949 6384 tablet_bootstrap.cc:492] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: No bootstrap required, opened a new log
I20260925 04:34:24.107116 6384 ts_tablet_manager.cc:1397] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 04:34:24.107156 6229 heartbeater.cc:503] Master 127.27.46.254:45745 was elected leader, sending a full tablet report...
I20260925 04:34:24.107362 6385 ts_tablet_manager.cc:1428] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260925 04:34:24.111343 6384 raft_consensus.cc:348] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.111562 6384 raft_consensus.cc:374] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:34:24.111621 6384 raft_consensus.cc:729] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd23807d56d49518b6e36485e150735, State: Initialized, Role: FOLLOWER
I20260925 04:34:24.111999 6384 consensus_queue.cc:260] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.112560 6365 heartbeater.cc:503] Master 127.27.46.254:45745 was elected leader, sending a full tablet report...
I20260925 04:34:24.112856 6384 ts_tablet_manager.cc:1428] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:34:24.197332 6390 raft_consensus.cc:482] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:34:24.197654 6390 raft_consensus.cc:504] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.198926 6390 leader_election.cc:288] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08c9eafcaea347be85ac054981ab0607 (127.27.46.193:44001), dbd23807d56d49518b6e36485e150735 (127.27.46.195:37653)
I20260925 04:34:24.203672 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.199375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:34719 (local address 127.27.46.195:37653)
0925 04:34:24.199736 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.199742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.199865 (+ 123us) server_negotiation.cc:407] Connection header received
0925 04:34:24.200055 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.200059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.200122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.200213 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:24.200494 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.201141 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.202521 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.203136 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.203245 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.203300 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.203460 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.203530 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260925 04:34:24.203672 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.199302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:44983 (local address 127.27.46.193:44001)
0925 04:34:24.199545 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.199550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.199570 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:34:24.199641 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.199645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.199701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.199784 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:34:24.200088 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.200878 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.202396 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.203025 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.203154 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.203228 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.203437 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.203520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 04:34:24.203673 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.199228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44001 (local address 127.27.46.194:44983)
0925 04:34:24.199419 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.199435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.199535 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.199808 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.199811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.199831 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:24.199959 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.199965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.200992 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.200995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.202261 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.202272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.203146 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.203148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.203277 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.203306 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.203386 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.203471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 04:34:24.203682 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.199283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:37653 (local address 127.27.46.194:34719)
0925 04:34:24.199783 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.199796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:34:24.199934 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:34:24.200222 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:34:24.200225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:34:24.200240 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:34:24.200360 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.200367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.201251 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.201254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.202413 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:34:24.202420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.203246 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.203249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:34:24.203333 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.203363 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:34:24.203428 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:34:24.203476 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":140,"threads_started":1}
I20260925 04:34:24.204520 6045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067b58054353478c9f0b0a48c70461b3" candidate_uuid: "ce17078744b644fb94db256077e82420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c9eafcaea347be85ac054981ab0607" is_pre_election: true
I20260925 04:34:24.204514 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067b58054353478c9f0b0a48c70461b3" candidate_uuid: "ce17078744b644fb94db256077e82420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd23807d56d49518b6e36485e150735" is_pre_election: true
I20260925 04:34:24.204859 6045 raft_consensus.cc:2454] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce17078744b644fb94db256077e82420 in term 0.
I20260925 04:34:24.204859 6319 raft_consensus.cc:2454] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce17078744b644fb94db256077e82420 in term 0.
I20260925 04:34:24.205394 6115 leader_election.cc:302] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08c9eafcaea347be85ac054981ab0607, ce17078744b644fb94db256077e82420, dbd23807d56d49518b6e36485e150735; no voters:
I20260925 04:34:24.205749 6390 raft_consensus.cc:2781] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:34:24.205853 6390 raft_consensus.cc:482] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:34:24.205930 6390 raft_consensus.cc:3037] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:24.207106 6390 raft_consensus.cc:504] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.207468 6390 leader_election.cc:288] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [CANDIDATE]: Term 1 election: Requested vote from peers 08c9eafcaea347be85ac054981ab0607 (127.27.46.193:44001), dbd23807d56d49518b6e36485e150735 (127.27.46.195:37653)
I20260925 04:34:24.207821 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067b58054353478c9f0b0a48c70461b3" candidate_uuid: "ce17078744b644fb94db256077e82420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd23807d56d49518b6e36485e150735"
I20260925 04:34:24.207858 6045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067b58054353478c9f0b0a48c70461b3" candidate_uuid: "ce17078744b644fb94db256077e82420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c9eafcaea347be85ac054981ab0607"
I20260925 04:34:24.207999 6319 raft_consensus.cc:3037] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:24.208024 6045 raft_consensus.cc:3037] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:34:24.209122 6319 raft_consensus.cc:2454] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce17078744b644fb94db256077e82420 in term 1.
I20260925 04:34:24.209268 6045 raft_consensus.cc:2454] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce17078744b644fb94db256077e82420 in term 1.
I20260925 04:34:24.209455 6115 leader_election.cc:302] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce17078744b644fb94db256077e82420, dbd23807d56d49518b6e36485e150735; no voters:
I20260925 04:34:24.209666 6390 raft_consensus.cc:2781] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:34:24.209924 6390 raft_consensus.cc:686] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 1 LEADER]: Becoming Leader. State: Replica: ce17078744b644fb94db256077e82420, State: Running, Role: LEADER
I20260925 04:34:24.210338 6390 consensus_queue.cc:237] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } }
I20260925 04:34:24.213636 5899 catalog_manager.cc:5495] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce17078744b644fb94db256077e82420 (127.27.46.194). New cstate: current_term: 1 leader_uuid: "ce17078744b644fb94db256077e82420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17078744b644fb94db256077e82420" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 } health_report { overall_health: UNKNOWN } } }
I20260925 04:34:24.238325 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.234644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.27.46.194:46045)
0925 04:34:24.234774 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:34:24.234778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:34:24.234793 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:34:24.234877 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:34:24.234880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:34:24.234939 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:34:24.235043 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:34:24.235269 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.235759 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.236647 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:34:24.236903 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:34:24.237018 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:34:24.237348 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:34:24.237892 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:34:24.238013 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:34:24.238094 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 04:34:24.238155 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:34:24.244155 6319 raft_consensus.cc:1261] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 1 FOLLOWER]: Refusing update from remote peer ce17078744b644fb94db256077e82420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:24.244155 6045 raft_consensus.cc:1261] T 067b58054353478c9f0b0a48c70461b3 P 08c9eafcaea347be85ac054981ab0607 [term 1 FOLLOWER]: Refusing update from remote peer ce17078744b644fb94db256077e82420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:34:24.244920 6393 consensus_queue.cc:1035] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd23807d56d49518b6e36485e150735" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:24.245153 6390 consensus_queue.cc:1035] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c9eafcaea347be85ac054981ab0607" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:34:24.251117 6396 mvcc.cc:205] Tried to move back new op lower bound from 7333113299936952320 to 7333113299808276480. Current Snapshot: MvccSnapshot[applied={T|T < 7333113299936952320}]
I20260925 04:34:24.253419 6397 mvcc.cc:205] Tried to move back new op lower bound from 7333113299936952320 to 7333113299808276480. Current Snapshot: MvccSnapshot[applied={T|T < 7333113299936952320}]
I20260925 04:34:24.260219 5898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:24.260360 5898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:34:24.263206 5898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32793012d70e4928bd07ff3fb014c2f4: Sending DeleteTablet for 3 replicas of tablet 067b58054353478c9f0b0a48c70461b3
I20260925 04:34:24.264043 6024 tablet_service.cc:1514] Processing DeleteTablet for tablet 067b58054353478c9f0b0a48c70461b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:24 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40076
I20260925 04:34:24.264142 6162 tablet_service.cc:1514] Processing DeleteTablet for tablet 067b58054353478c9f0b0a48c70461b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:24 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47344
I20260925 04:34:24.265065 6407 tablet_replica.cc:330] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: stopping tablet replica
I20260925 04:34:24.265395 6407 raft_consensus.cc:2229] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:34:24.265684 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5961
I20260925 04:34:24.266776 6407 raft_consensus.cc:2258] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:24.267452 6299 tablet_service.cc:1514] Processing DeleteTablet for tablet 067b58054353478c9f0b0a48c70461b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:34:24 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:53258
I20260925 04:34:24.268813 6407 ts_tablet_manager.cc:1905] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:34:24.272105 6407 ts_tablet_manager.cc:1918] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:34:24.272223 6407 log.cc:1196] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/067b58054353478c9f0b0a48c70461b3
I20260925 04:34:24.272519 6407 ts_tablet_manager.cc:1939] T 067b58054353478c9f0b0a48c70461b3 P ce17078744b644fb94db256077e82420: Deleting consensus metadata
I20260925 04:34:24.273682 5883 catalog_manager.cc:4841] TS ce17078744b644fb94db256077e82420 (127.27.46.194:46045): tablet 067b58054353478c9f0b0a48c70461b3 (table test-table [id=f07f4b9245484a779f36cd09b404e80c]) successfully deleted
I20260925 04:34:24.277796 6408 tablet_replica.cc:330] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: stopping tablet replica
W20260925 04:34:24.278185 5886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:44001 (error 108)
I20260925 04:34:24.278247 6408 raft_consensus.cc:2229] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:34:24.278648 6408 raft_consensus.cc:2258] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:34:24.279392 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:34:24.279098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44001 (local address 127.0.0.1:40080)
0925 04:34:24.279217 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:34:24.279290 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 04:34:24.279392 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 6096
I20260925 04:34:24.280632 6408 ts_tablet_manager.cc:1905] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:34:24.280997 5886 catalog_manager.cc:4584] TS 08c9eafcaea347be85ac054981ab0607 (127.27.46.193:44001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 067b58054353478c9f0b0a48c70461b3: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44001: connect: Connection refused (error 111)
I20260925 04:34:24.283905 6408 ts_tablet_manager.cc:1918] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:34:24.284017 6408 log.cc:1196] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/067b58054353478c9f0b0a48c70461b3
I20260925 04:34:24.284365 6408 ts_tablet_manager.cc:1939] T 067b58054353478c9f0b0a48c70461b3 P dbd23807d56d49518b6e36485e150735: Deleting consensus metadata
I20260925 04:34:24.285917 5884 catalog_manager.cc:4841] TS dbd23807d56d49518b6e36485e150735 (127.27.46.195:37653): tablet 067b58054353478c9f0b0a48c70461b3 (table test-table [id=f07f4b9245484a779f36cd09b404e80c]) successfully deleted
I20260925 04:34:24.287566 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 6234
I20260925 04:34:24.294409 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 5867
2026-09-25T04:34:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2363 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:34:24 dist-test-slave-r8r3 krb5kdc[6412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:34:24 dist-test-slave-r8r3 krb5kdc[6412](info): set up 2 sockets
Sep 25 04:34:24 dist-test-slave-r8r3 krb5kdc[6412](info): commencing operation
krb5kdc: starting...
W20260925 04:34:25.342754 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:34:25 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:34:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangeradmin_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangeradmin_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangerlookup_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangerlookup_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
I20260925 04:34:25.513092 27835 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjz8mnA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/postgres -l logfile start
2026-09-25 04:34:28.035 UTC [6443] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:34:28.036 UTC [6443] LOG: listening on IPv4 address "127.27.46.212", port 54471
2026-09-25 04:34:28.039 UTC [6443] LOG: listening on Unix socket "/tmp/.s.PGSQL.54471"
2026-09-25 04:34:28.060 UTC [6446] LOG: database system was shut down at 2026-09-25 04:34:26 UTC
2026-09-25 04:34:28.064 UTC [6443] LOG: database system is ready to accept connections
I20260925 04:34:29.026733 27835 mini_postgres.cc:96] Postgres bound to 54471
2026-09-25 04:34:29.038 UTC [6454] FATAL: database "slave" does not exist
127.27.46.212:54471 - accepting connections
I20260925 04:34:29.038772 27835 mini_ranger.cc:161] Starting Ranger...
I20260925 04:34:29.058833 27835 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:34:29.534057 27835 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:34:29.534155 27835 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin
2026-09-25 04:34:29,847 [I] DB FLAVOR :POSTGRES
2026-09-25 04:34:29,847 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:34:29,847 [I] Checking connection..
2026-09-25 04:34:29,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:34:30,075 [I] Checking connection passed.
2026-09-25 04:34:30,076 [I] --------- Verifying version history table ---------
2026-09-25 04:34:30,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:34:30,320 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:34:30,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:34:30,585 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:34:30,585 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:34:30,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:34:30.814 UTC [6518] WARNING: there is no transaction in progress
2026-09-25 04:34:30,820 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:34:30,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:34:31,069 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:34:31,069 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:34:31,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:34:31,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:34:31,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r8r3.c.gcp-upstream.internal','N') ;"
2026-09-25 04:34:31,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:34:32,073 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:34:32,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:34:32,335 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:34:32,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:34:32,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:34:32,852 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:34:32,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:34:33.897 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:33.915 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:33.933 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:33.950 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:33.967 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:33.987 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.046 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.053 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.069 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.080 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.091 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.104 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.114 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.121 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.127 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34.134 UTC [6629] WARNING: there is no transaction in progress
2026-09-25 04:34:34,784 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:34:34,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:34:35,056 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:34:35,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:34:35,314 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:34:35,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:34:35,557 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:34:35,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:34:35,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:34:36,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r8r3.c.gcp-upstream.internal';"
2026-09-25 04:34:36,309 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:34:36,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:34:36,572 [I] DB_PATCHES have already been applied
I20260925 04:34:36.577764 27835 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjz8mnA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:34:36.577842 27835 mini_ranger.cc:193] Using host: 127.27.46.212
I20260925 04:34:36.581576 27835 mini_ranger.cc:227] Ranger admin URL: http://127.27.46.212:44139
Sep 25, 2026 4:34:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:34:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:34:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:34:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:34:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:34:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:34:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:34:37.695308 27835 mini_ranger.cc:160] Time spent starting Ranger: real 8.657s user 0.005s sys 0.005s
Sep 25, 2026 4:34:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:34:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:34:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:34:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:34:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:34:58.795907 27835 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangerkms_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/rangerkms_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:34:58 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:34:58.899223 27835 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:34:58.909870 27835 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:34:59.805544 27835 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:34:59.805653 27835 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms
I20260925 04:34:59.805766 27835 mini_ranger_kms.cc:227] Using postgres at 127.27.46.212:54471
2026-09-25 04:35:00,599 [I] DB FLAVOR :POSTGRES
2026-09-25 04:35:00,599 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:35:00,599 [I] Checking connection
2026-09-25 04:35:00,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:35:00,866 [I] connection success
2026-09-25 04:35:00,866 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:35:00,866 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:35:00,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:35:01,097 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:35:01,098 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:35:01,098 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:35:01,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:54471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:35:01,350 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:35:01.818774 27835 mini_ranger_kms.cc:310] Created kms service
I20260925 04:35:02.009925 27835 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:35:02.056535 27835 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:35:02 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:02.491104 27835 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:35:02.491264 27835 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:35:02.491286 27835 mini_ranger_kms.cc:240] Using host: 127.27.46.212
I20260925 04:35:02.495349 27835 mini_ranger_kms.cc:276] Ranger KMS PID: 6810
I20260925 04:35:02.495504 27835 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.46.212:34543
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:35:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 04:35:04 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790310904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:04.799465 27835 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.900s user 0.002s sys 0.391s
I20260925 04:35:04.799600 27835 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:35:04.799722 27835 mini_ranger_kms.cc:401] 127.27.46.212:34543/kms/v1/keys
Sep 25 04:35:05 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
Sep 25 04:35:05 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
Sep 25 04:35:07 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:35:07.970069 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45753
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:45753
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:34543/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:08.108609 6861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:08.108903 6861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:08.108946 6861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:08.113127 6861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:35:08.113195 6861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:08.113219 6861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:08.113237 6861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:35:08.113301 6861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:35:08.119035 6861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.46.212:34543/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:45753
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:08.120189 6861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:08.121330 6861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:08.131037 6867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:08.131239 6869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:08.131052 6866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:08.131579 6861 server_base.cc:1034] running on GCE node
I20260925 04:35:08.132184 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:08.133436 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:08.134662 6861 hybrid_clock.cc:648] HybridClock initialized: now 1790310908134618 us; error 58 us; skew 500 ppm
Sep 25 04:35:08 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310908, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:08.137704 6861 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:35:08.138628 6861 webserver.cc:460] Webserver started at http://127.27.46.254:38225/ using document root <none> and password file <none>
I20260925 04:35:08.139227 6861 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:08.139325 6861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:08.139575 6861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:08 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310908, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:08.190802 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "442c7e31793c467bad3a792af6421363"
format_stamp: "Formatted at 2026-09-25 04:35:08 on dist-test-slave-r8r3"
server_key: "e78c50a26100cdbd87dbcdd8e400855b"
server_key_iv: "814cc65ed1e64abe5e21127ab0921720"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:08.191478 6861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "442c7e31793c467bad3a792af6421363"
format_stamp: "Formatted at 2026-09-25 04:35:08 on dist-test-slave-r8r3"
server_key: "e78c50a26100cdbd87dbcdd8e400855b"
server_key_iv: "814cc65ed1e64abe5e21127ab0921720"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:08.195851 6861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:35:08.212776 6876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:08.214655 6861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:35:08.214812 6861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "442c7e31793c467bad3a792af6421363"
format_stamp: "Formatted at 2026-09-25 04:35:08 on dist-test-slave-r8r3"
server_key: "e78c50a26100cdbd87dbcdd8e400855b"
server_key_iv: "814cc65ed1e64abe5e21127ab0921720"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:08.214965 6861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:08.245205 6861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:08.248909 6861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:08.249226 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:08.259336 6861 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:45753
I20260925 04:35:08.259397 6929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:45753 every 8 connection(s)
I20260925 04:35:08.260608 6861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:35:08.264781 6930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:08.269445 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 6861
I20260925 04:35:08.269644 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:35:08.269975 27835 external_mini_cluster.cc:1428] Setting key cda67a884b2ae797adf1e7f2ce2aaf71
I20260925 04:35:08.273072 6930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: Bootstrap starting.
I20260925 04:35:08.275923 6930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:08.276952 6930 log.cc:824] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:08.279395 6930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: No bootstrap required, opened a new log
Sep 25 04:35:08 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:08.283835 6930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } }
I20260925 04:35:08.284088 6930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:08.284162 6930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442c7e31793c467bad3a792af6421363, State: Initialized, Role: FOLLOWER
I20260925 04:35:08.284708 6930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } }
I20260925 04:35:08.284938 6930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:35:08.285020 6930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:35:08.285130 6930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:08.286185 6930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } }
I20260925 04:35:08.286581 6930 leader_election.cc:302] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 442c7e31793c467bad3a792af6421363; no voters:
I20260925 04:35:08.286980 6930 leader_election.cc:288] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:35:08.287609 6937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:35:08.287859 6937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [term 1 LEADER]: Becoming Leader. State: Replica: 442c7e31793c467bad3a792af6421363, State: Running, Role: LEADER
I20260925 04:35:08.288218 6937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } }
I20260925 04:35:08.288448 6930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:35:08.289678 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "442c7e31793c467bad3a792af6421363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } } }
I20260925 04:35:08.290006 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:08.290371 6937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 442c7e31793c467bad3a792af6421363. Latest consensus state: current_term: 1 leader_uuid: "442c7e31793c467bad3a792af6421363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "442c7e31793c467bad3a792af6421363" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45753 } } }
I20260925 04:35:08.290597 6937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363 [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:08.290757 6933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:08.271499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.27.46.254:45753)
0925 04:35:08.271939 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:08.271961 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:08.273915 (+ 1954us) server_negotiation.cc:407] Connection header received
0925 04:35:08.275022 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:08.275047 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:08.275496 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:08.275872 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:08.276907 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:08.278428 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:08.279831 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:08.280530 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:08.280694 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:08.284197 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:08.284205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:08.284225 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:08.284260 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:08.286442 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:08.286967 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:08.286974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:08.286981 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:08.287153 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:08.287509 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:08.287513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:08.287515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:08.288128 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:08.289185 (+ 1057us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:08.289352 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:35:08.289703 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":148,"threads_started":1}
I20260925 04:35:08.291400 6930 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:35:08.297087 6945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:35:08.302903 6945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:35:08.311208 6945 catalog_manager.cc:1347] Generated new cluster ID: 96edf456935b49fb9b986be63c30573d
I20260925 04:35:08.311389 6945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:35:08.326814 6945 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:35:08.328234 6945 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:35:08.335042 6945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: Generated new TSK 0
I20260925 04:35:08.336010 6945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:35:09.096951 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:34543/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:09.255924 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:09.256203 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:09.256278 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:09.261165 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:09.261267 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:09.261368 6977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:35:09.268962 6977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.46.212:34543/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:09.270073 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:09.271051 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:09.283028 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:09.283036 6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:09.283444 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:09.283891 6977 server_base.cc:1034] running on GCE node
I20260925 04:35:09.284343 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:09.285149 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:09.286330 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790310909286273 us; error 69 us; skew 500 ppm
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:09.289434 6977 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:35:09.290587 6977 webserver.cc:460] Webserver started at http://127.27.46.193:43333/ using document root <none> and password file <none>
I20260925 04:35:09.291190 6977 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:09.291281 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:09.291500 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:09.309459 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "f4be5ad5f60c459287e62b15e6738b7f"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "310c437b71eaa40ef4c422f3b1bed0a9"
server_key_iv: "45dec889d7e479083dde002094979ba4"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.310312 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "f4be5ad5f60c459287e62b15e6738b7f"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "310c437b71eaa40ef4c422f3b1bed0a9"
server_key_iv: "45dec889d7e479083dde002094979ba4"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.323652 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260925 04:35:09.342307 6994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.344089 6977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:35:09.344235 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "f4be5ad5f60c459287e62b15e6738b7f"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "310c437b71eaa40ef4c422f3b1bed0a9"
server_key_iv: "45dec889d7e479083dde002094979ba4"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.344375 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:09.364399 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:09.368291 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:09.368551 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:09.369278 6977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:09.370410 6977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:09.370517 6977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.370643 6977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:09.370730 6977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.391417 6977 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:42953
I20260925 04:35:09.392678 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:35:09.396335 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 6977
I20260925 04:35:09.396471 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:35:09.396764 27835 external_mini_cluster.cc:1428] Setting key 1b2669515bc08e24deee08d99b94fa83
I20260925 04:35:09.398574 7112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:42953 every 8 connection(s)
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:09.414248 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:09.397998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:52703 (local address 127.27.46.254:45753)
0925 04:35:09.400893 (+ 2895us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:09.400898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:09.400914 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:35:09.400957 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:09.400960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:09.401010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:09.401101 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:09.402780 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.403674 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.405613 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.405985 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.406069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:09.409269 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:09.409275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:09.409279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:09.409327 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:09.411380 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:09.412103 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:09.412107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:09.412109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:09.412207 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:09.412684 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:09.412689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:09.412691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:09.412982 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:09.413084 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:09.413921 (+ 837us) server_negotiation.cc:299] Negotiation successful
0925 04:35:09.414059 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2814,"thread_start_us":78,"threads_started":1}
I20260925 04:35:09.415988 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:09.396890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45753 (local address 127.27.46.193:52703)
0925 04:35:09.399263 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:09.399329 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:09.400455 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:09.401532 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:09.401543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:09.401906 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:09.402595 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:09.402617 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.403835 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.403839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:09.405446 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:09.405457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.406291 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.406295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:09.406510 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:09.407226 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:09.407247 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:09.409041 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:09.411537 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:09.411546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:09.411566 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:09.411965 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:09.412343 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:09.412347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:09.412349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:09.412555 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:09.413165 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:09.413175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:09.413613 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 04:35:09.414506 (+ 893us) client_negotiation.cc:240] Negotiation successful
0925 04:35:09.414863 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":128,"threads_started":1}
I20260925 04:35:09.417510 7113 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45753
I20260925 04:35:09.417775 7113 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:09.418409 7113 heartbeater.cc:511] Master 127.27.46.254:45753 requested a full tablet report, sending...
I20260925 04:35:09.420229 6893 ts_manager.cc:194] Registered new tserver with Master: f4be5ad5f60c459287e62b15e6738b7f (127.27.46.193:42953)
I20260925 04:35:09.421903 6893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:52703
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:35:09.451262 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:34543/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:35:09.619553 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:09.620059 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:09.620157 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:09.626560 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:09.626677 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:09.626822 7119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:35:09.635356 7119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.46.212:34543/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:09.636895 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:09.638342 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:09.653416 7126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:09.653982 7125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:09.654718 7128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:09.655128 7119 server_base.cc:1034] running on GCE node
I20260925 04:35:09.655610 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:09.656447 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:09.657640 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790310909657571 us; error 82 us; skew 500 ppm
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:09.661391 7119 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:35:09.662508 7119 webserver.cc:460] Webserver started at http://127.27.46.194:33325/ using document root <none> and password file <none>
I20260925 04:35:09.663244 7119 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:09.663365 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:09.663733 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:09.684068 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "57fa5227c6524fcb92339987ee18c9ba"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "64425a1d6c12a83c15bbc948c4899995"
server_key_iv: "87e32b885f29389516de59c61f9af087"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.684827 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "57fa5227c6524fcb92339987ee18c9ba"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "64425a1d6c12a83c15bbc948c4899995"
server_key_iv: "87e32b885f29389516de59c61f9af087"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.690038 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:35:09.713973 7135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.715662 7119 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260925 04:35:09.715821 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "57fa5227c6524fcb92339987ee18c9ba"
format_stamp: "Formatted at 2026-09-25 04:35:09 on dist-test-slave-r8r3"
server_key: "64425a1d6c12a83c15bbc948c4899995"
server_key_iv: "87e32b885f29389516de59c61f9af087"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:09.715974 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:09.744096 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:09.748018 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:09.748207 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:09.748981 7119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:09.750295 7119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:09.750509 7119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.750581 7119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:09.750617 7119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:09.762838 7119 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:45237
I20260925 04:35:09.764032 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:35:09.770943 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 7119
I20260925 04:35:09.771073 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:35:09.771370 27835 external_mini_cluster.cc:1428] Setting key 4e687037463882163f91e362eea3b3bf
I20260925 04:35:09.770274 7250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:45237 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Sep 25 04:35:09 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:35:09.809370 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:09.773029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:35857 (local address 127.27.46.254:45753)
0925 04:35:09.773148 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:09.773153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:09.782558 (+ 9405us) server_negotiation.cc:407] Connection header received
0925 04:35:09.788435 (+ 5877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:09.788440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:09.788507 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:09.788565 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:09.790789 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.791628 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.793489 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.793867 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.793956 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:09.804036 (+ 10080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:09.804043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:09.804046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:09.804133 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:09.806122 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:09.807205 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:09.807210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:09.807212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:09.807325 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:09.807795 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:09.807801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:09.807803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:09.808052 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:09.808144 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:09.808966 (+ 822us) server_negotiation.cc:299] Negotiation successful
0925 04:35:09.809158 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:35:09.812108 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:09.773320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45753 (local address 127.27.46.194:35857)
0925 04:35:09.782352 (+ 9032us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:09.782433 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:09.788108 (+ 5675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:09.789205 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:09.789216 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:09.789678 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:09.790605 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:09.790626 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.791876 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.791879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:09.793338 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:09.793351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:09.794254 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:09.794259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:09.794541 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:09.795328 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:09.795352 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:09.801341 (+ 5989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:09.806521 (+ 5180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:09.806530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:09.806563 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:09.807068 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:09.807454 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:09.807457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:09.807459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:09.807672 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:09.808218 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:09.808226 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:09.808620 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 04:35:09.810297 (+ 1677us) client_negotiation.cc:240] Negotiation successful
0925 04:35:09.810721 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8759,"thread_start_us":192,"threads_started":1}
I20260925 04:35:09.817981 7251 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45753
I20260925 04:35:09.818305 7251 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:09.818991 7251 heartbeater.cc:511] Master 127.27.46.254:45753 requested a full tablet report, sending...
I20260925 04:35:09.820518 6893 ts_manager.cc:194] Registered new tserver with Master: 57fa5227c6524fcb92339987ee18c9ba (127.27.46.194:45237)
I20260925 04:35:09.821369 6893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:35857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:35:09.848809 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:34543/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:43425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:10.046168 7256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:10.046635 7256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:10.046782 7256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:10.052994 7256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:10.053157 7256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:10.053330 7256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:35:10.061632 7256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.46.212:34543/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:10.063282 7256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:10.064783 7256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:10.077992 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:10.079208 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:10.079655 7256 server_base.cc:1034] running on GCE node
W20260925 04:35:10.081633 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:10.082291 7256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:10.083118 7256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:10.084314 7256 hybrid_clock.cc:648] HybridClock initialized: now 1790310910084272 us; error 52 us; skew 500 ppm
Sep 25 04:35:10 dist-test-slave-r8r3 krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310910, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:10.087955 7256 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:35:10.089010 7256 webserver.cc:460] Webserver started at http://127.27.46.195:41793/ using document root <none> and password file <none>
I20260925 04:35:10.089855 7256 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:10.089969 7256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:10.090274 7256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:10 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310910, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
Sep 25 04:35:10 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:10.106055 7256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "d8c105d8692a48f192be5479c0ebd5f6"
format_stamp: "Formatted at 2026-09-25 04:35:10 on dist-test-slave-r8r3"
server_key: "9b7ad65a90c761421a16c62ae8d057f2"
server_key_iv: "a64a47eaa8c25f6c269920cd5aa9b634"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:10.107226 7256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c105d8692a48f192be5479c0ebd5f6"
format_stamp: "Formatted at 2026-09-25 04:35:10 on dist-test-slave-r8r3"
server_key: "9b7ad65a90c761421a16c62ae8d057f2"
server_key_iv: "a64a47eaa8c25f6c269920cd5aa9b634"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:10.115141 7256 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.004s
I20260925 04:35:10.161729 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:10.163655 7256 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260925 04:35:10.163807 7256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "d8c105d8692a48f192be5479c0ebd5f6"
format_stamp: "Formatted at 2026-09-25 04:35:10 on dist-test-slave-r8r3"
server_key: "9b7ad65a90c761421a16c62ae8d057f2"
server_key_iv: "a64a47eaa8c25f6c269920cd5aa9b634"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:10.163964 7256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:10.191303 7256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:10.195007 7256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:10.195201 7256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:10.195876 7256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:10.196894 7256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:10.196954 7256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:10.197018 7256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:10.197047 7256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:10.209823 7256 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:41991
I20260925 04:35:10.211076 7256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:35:10.213070 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 7256
I20260925 04:35:10.213214 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:35:10.213747 27835 external_mini_cluster.cc:1428] Setting key b150fc70baed4b68303cec00c2fa7dd8
I20260925 04:35:10.223048 7386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:41991 every 8 connection(s)
Sep 25 04:35:10 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310910, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:10.243921 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.225718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:42541 (local address 127.27.46.254:45753)
0925 04:35:10.227168 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.227173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.228853 (+ 1680us) server_negotiation.cc:407] Connection header received
0925 04:35:10.230379 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.230383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.230436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.230502 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:10.232148 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.232969 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.234688 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.235098 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.235177 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.238202 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:10.238206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:10.238209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:10.238233 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:10.241454 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:10.242091 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:10.242094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:10.242096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:10.242180 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:10.242592 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:10.242595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:10.242597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:10.242805 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:10.242883 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.243625 (+ 742us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.243754 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1354}
I20260925 04:35:10.245595 7389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.226061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45753 (local address 127.27.46.195:42541)
0925 04:35:10.228690 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.228755 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.230103 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.230919 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.230928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.231321 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:10.231984 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.232003 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.233128 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.233132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.234551 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.234562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.235392 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.235396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.235587 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.236273 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:10.236308 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:10.238025 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:10.241629 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:10.241635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:10.241650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:10.241978 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:10.242296 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:10.242299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:10.242301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:10.242477 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:10.242945 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:10.242952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:10.243245 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.244134 (+ 889us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.244480 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"thread_start_us":174,"threads_started":1}
I20260925 04:35:10.246807 7387 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45753
I20260925 04:35:10.247061 7387 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:10.247601 7387 heartbeater.cc:511] Master 127.27.46.254:45753 requested a full tablet report, sending...
I20260925 04:35:10.248684 6893 ts_manager.cc:194] Registered new tserver with Master: d8c105d8692a48f192be5479c0ebd5f6 (127.27.46.195:41991)
I20260925 04:35:10.249423 6893 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:42541
I20260925 04:35:10.253170 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:35:10.266665 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.256231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.27.46.254:45753)
0925 04:35:10.256352 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.256357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.256904 (+ 547us) server_negotiation.cc:407] Connection header received
0925 04:35:10.257074 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.257077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.257121 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.257167 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:10.258219 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.259008 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.260274 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.260618 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.260691 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.262564 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:10.262569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:10.262573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:10.262603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:10.264463 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:10.265026 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:10.265030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:10.265031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:10.265132 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:10.265521 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:10.265526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:10.265528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:10.265775 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:10.265896 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.266325 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.266458 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:35:10.271348 6893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:35:10.276933 6893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:35:10.305343 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.293374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:42953 (local address 127.0.0.1:44774)
0925 04:35:10.297194 (+ 3820us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.297207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.297310 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.299061 (+ 1751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.299067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.299094 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:10.299287 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.299316 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.301719 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.301723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.303206 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.303217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.304700 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.304705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.304847 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.304902 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.305018 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.305165 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3760,"thread_start_us":711,"threads_started":1}
I20260925 04:35:10.306072 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.293758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.27.46.193:42953)
0925 04:35:10.297045 (+ 3287us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.297064 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.297675 (+ 611us) server_negotiation.cc:407] Connection header received
0925 04:35:10.297788 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.297794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.298017 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.300165 (+ 2148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:10.300330 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.301568 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.303480 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.304345 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.304465 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.304629 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.305757 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.305896 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3174,"thread_start_us":133,"threads_started":1}
I20260925 04:35:10.307199 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.290720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:41991 (local address 127.0.0.1:35348)
0925 04:35:10.296541 (+ 5821us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.296595 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.296729 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.299978 (+ 3249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.299981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.300001 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:10.300150 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.300158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.302549 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.302553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.303578 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.303589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.304918 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.304922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.305099 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.305126 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.305201 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.305271 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5635,"thread_start_us":96,"threads_started":1}
I20260925 04:35:10.308076 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.291974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.27.46.195:41991)
0925 04:35:10.299514 (+ 7540us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.299550 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.299572 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:35:10.299637 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.299641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.299795 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.300011 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:10.300319 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.302393 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.303778 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.304586 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.304699 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.304854 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.307620 (+ 2766us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.307923 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7418,"thread_start_us":196,"threads_started":1}
I20260925 04:35:10.309546 7042 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdd47b394144c46946a09abe0aa777f (DEFAULT_TABLE table=test-table [id=b1ef3515e0294ee39228059735c0ff80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:10.311547 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.294658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:45237 (local address 127.0.0.1:52002)
0925 04:35:10.305728 (+ 11070us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.305743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.305988 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.306791 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.306804 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.306832 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:10.306945 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.306959 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.308503 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.308506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.309690 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.309701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.311132 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.311147 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.311249 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.311273 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.311351 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.311418 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10986,"thread_start_us":77,"threads_started":1}
I20260925 04:35:10.311236 7320 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdd47b394144c46946a09abe0aa777f (DEFAULT_TABLE table=test-table [id=b1ef3515e0294ee39228059735c0ff80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:10.311992 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.295048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.27.46.194:45237)
0925 04:35:10.297613 (+ 2565us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.297631 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.305905 (+ 8274us) server_negotiation.cc:407] Connection header received
0925 04:35:10.306149 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.306155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.306398 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.306569 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:10.307088 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.308343 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.309944 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.310765 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.310867 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.311023 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.311714 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.311832 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2395,"thread_start_us":73,"threads_started":1}
I20260925 04:35:10.312273 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdd47b394144c46946a09abe0aa777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:10.314606 7184 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdd47b394144c46946a09abe0aa777f (DEFAULT_TABLE table=test-table [id=b1ef3515e0294ee39228059735c0ff80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:10.315588 7184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdd47b394144c46946a09abe0aa777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:10.312040 7042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdd47b394144c46946a09abe0aa777f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:10.327509 7406 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Bootstrap starting.
I20260925 04:35:10.330677 7407 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Bootstrap starting.
I20260925 04:35:10.331018 7406 tablet_bootstrap.cc:654] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:10.332007 7406 log.cc:824] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:10.332947 7408 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Bootstrap starting.
I20260925 04:35:10.334280 7407 tablet_bootstrap.cc:654] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:10.335696 7408 tablet_bootstrap.cc:654] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:10.335779 7407 log.cc:824] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:10.336735 7408 log.cc:824] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:10.338186 7407 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: No bootstrap required, opened a new log
I20260925 04:35:10.338403 7407 ts_tablet_manager.cc:1397] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 04:35:10.338887 7406 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: No bootstrap required, opened a new log
I20260925 04:35:10.339071 7406 ts_tablet_manager.cc:1397] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260925 04:35:10.343612 7406 raft_consensus.cc:348] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.344051 7406 raft_consensus.cc:374] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:10.343923 7407 raft_consensus.cc:348] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.344187 7407 raft_consensus.cc:374] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:10.344247 7407 raft_consensus.cc:729] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fa5227c6524fcb92339987ee18c9ba, State: Initialized, Role: FOLLOWER
I20260925 04:35:10.344224 7406 raft_consensus.cc:729] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c105d8692a48f192be5479c0ebd5f6, State: Initialized, Role: FOLLOWER
I20260925 04:35:10.344825 7406 consensus_queue.cc:260] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.344808 7407 consensus_queue.cc:260] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.346127 7406 ts_tablet_manager.cc:1428] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:35:10.346127 7407 ts_tablet_manager.cc:1428] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:35:10.346534 7387 heartbeater.cc:503] Master 127.27.46.254:45753 was elected leader, sending a full tablet report...
I20260925 04:35:10.349056 7408 tablet_bootstrap.cc:492] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: No bootstrap required, opened a new log
I20260925 04:35:10.349298 7408 ts_tablet_manager.cc:1397] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260925 04:35:10.352072 7251 heartbeater.cc:503] Master 127.27.46.254:45753 was elected leader, sending a full tablet report...
I20260925 04:35:10.354144 7408 raft_consensus.cc:348] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.354488 7408 raft_consensus.cc:374] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:10.354591 7408 raft_consensus.cc:729] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4be5ad5f60c459287e62b15e6738b7f, State: Initialized, Role: FOLLOWER
I20260925 04:35:10.355116 7408 consensus_queue.cc:260] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.356539 7408 ts_tablet_manager.cc:1428] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260925 04:35:10.356935 7113 heartbeater.cc:503] Master 127.27.46.254:45753 was elected leader, sending a full tablet report...
I20260925 04:35:10.390558 7414 raft_consensus.cc:482] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:35:10.390865 7414 raft_consensus.cc:504] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.392861 7414 leader_election.cc:288] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c105d8692a48f192be5479c0ebd5f6 (127.27.46.195:41991), 57fa5227c6524fcb92339987ee18c9ba (127.27.46.194:45237)
I20260925 04:35:10.398222 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.392672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:49243 (local address 127.27.46.195:41991)
0925 04:35:10.392842 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.392849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.393193 (+ 344us) server_negotiation.cc:407] Connection header received
0925 04:35:10.393405 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.393408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.393459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.393532 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:10.394189 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.394962 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.396873 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.397448 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.397552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.397607 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.398003 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.398090 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:35:10.398677 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.392472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:41991 (local address 127.27.46.193:49243)
0925 04:35:10.393100 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.393115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.393273 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.393680 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.393683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.393728 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:10.394052 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.394060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.395232 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.395235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.396715 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.396726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.397716 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.397720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.397818 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.397856 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.398426 (+ 570us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.398508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":136,"threads_started":1}
W20260925 04:35:10.399269 7114 tablet.cc:2369] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:35:10.399308 7330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdd47b394144c46946a09abe0aa777f" candidate_uuid: "f4be5ad5f60c459287e62b15e6738b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c105d8692a48f192be5479c0ebd5f6" is_pre_election: true
I20260925 04:35:10.399597 7330 raft_consensus.cc:2454] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4be5ad5f60c459287e62b15e6738b7f in term 0.
I20260925 04:35:10.400116 6997 leader_election.cc:302] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8c105d8692a48f192be5479c0ebd5f6, f4be5ad5f60c459287e62b15e6738b7f; no voters:
I20260925 04:35:10.400403 7414 raft_consensus.cc:2781] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:35:10.400485 7414 raft_consensus.cc:482] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:35:10.400557 7414 raft_consensus.cc:3037] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:10.401926 7414 raft_consensus.cc:504] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.402400 7414 leader_election.cc:288] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [CANDIDATE]: Term 1 election: Requested vote from peers d8c105d8692a48f192be5479c0ebd5f6 (127.27.46.195:41991), 57fa5227c6524fcb92339987ee18c9ba (127.27.46.194:45237)
I20260925 04:35:10.402923 7330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdd47b394144c46946a09abe0aa777f" candidate_uuid: "f4be5ad5f60c459287e62b15e6738b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c105d8692a48f192be5479c0ebd5f6"
I20260925 04:35:10.403053 7330 raft_consensus.cc:3037] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:10.404379 7330 raft_consensus.cc:2454] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4be5ad5f60c459287e62b15e6738b7f in term 1.
I20260925 04:35:10.404718 6997 leader_election.cc:302] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8c105d8692a48f192be5479c0ebd5f6, f4be5ad5f60c459287e62b15e6738b7f; no voters:
I20260925 04:35:10.404940 7414 raft_consensus.cc:2781] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:35:10.405138 7414 raft_consensus.cc:686] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [term 1 LEADER]: Becoming Leader. State: Replica: f4be5ad5f60c459287e62b15e6738b7f, State: Running, Role: LEADER
I20260925 04:35:10.405586 7414 consensus_queue.cc:237] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } }
I20260925 04:35:10.408908 6893 catalog_manager.cc:5495] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f reported cstate change: term changed from 0 to 1, leader changed from <none> to f4be5ad5f60c459287e62b15e6738b7f (127.27.46.193). New cstate: current_term: 1 leader_uuid: "f4be5ad5f60c459287e62b15e6738b7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4be5ad5f60c459287e62b15e6738b7f" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 42953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20260925 04:35:10.419317 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.393682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:58751 (local address 127.27.46.194:45237)
0925 04:35:10.393862 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.393867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.394153 (+ 286us) server_negotiation.cc:407] Connection header received
0925 04:35:10.413157 (+ 19004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.413162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.413237 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.413300 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:10.415736 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.416500 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.417931 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.418491 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.418584 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.418636 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.418974 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.419192 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:35:10.419741 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.393222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:45237 (local address 127.27.46.193:58751)
0925 04:35:10.394062 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.394077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:10.394231 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:10.415450 (+ 21219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:10.415454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:10.415473 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:10.415564 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.415570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.416650 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.416653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.417778 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:10.417788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.418731 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.418734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:10.418819 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.418842 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:35:10.419465 (+ 623us) client_negotiation.cc:240] Negotiation successful
0925 04:35:10.419606 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":94,"threads_started":1}
I20260925 04:35:10.421126 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdd47b394144c46946a09abe0aa777f" candidate_uuid: "f4be5ad5f60c459287e62b15e6738b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fa5227c6524fcb92339987ee18c9ba"
I20260925 04:35:10.421156 7204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdd47b394144c46946a09abe0aa777f" candidate_uuid: "f4be5ad5f60c459287e62b15e6738b7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fa5227c6524fcb92339987ee18c9ba" is_pre_election: true
I20260925 04:35:10.421350 7203 raft_consensus.cc:3037] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:10.422998 7203 raft_consensus.cc:2454] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4be5ad5f60c459287e62b15e6738b7f in term 1.
I20260925 04:35:10.430109 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.424668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.27.46.193:42953)
0925 04:35:10.424808 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:10.424813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:10.424829 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:35:10.424906 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:10.424909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:10.424977 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:10.425090 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:35:10.425561 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.426295 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.427807 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:10.428137 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:10.428254 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:10.428775 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:35:10.429290 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:35:10.429413 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:10.429887 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 04:35:10.429951 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:35:10.438073 7203 raft_consensus.cc:1261] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 1 FOLLOWER]: Refusing update from remote peer f4be5ad5f60c459287e62b15e6738b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:35:10.439003 7330 raft_consensus.cc:1261] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 1 FOLLOWER]: Refusing update from remote peer f4be5ad5f60c459287e62b15e6738b7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:35:10.440037 7414 consensus_queue.cc:1035] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fa5227c6524fcb92339987ee18c9ba" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:35:10.440367 7418 consensus_queue.cc:1035] T 0fdd47b394144c46946a09abe0aa777f P f4be5ad5f60c459287e62b15e6738b7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8c105d8692a48f192be5479c0ebd5f6" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 41991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:35:10.453703 7423 mvcc.cc:205] Tried to move back new op lower bound from 7333113489143529472 to 7333113489024352256. Current Snapshot: MvccSnapshot[applied={T|T < 7333113489143529472}]
I20260925 04:35:10.454485 7424 mvcc.cc:205] Tried to move back new op lower bound from 7333113489143529472 to 7333113489024352256. Current Snapshot: MvccSnapshot[applied={T|T < 7333113489143529472}]
I20260925 04:35:10.464169 6892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:35:10.464517 6892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:35:10.467548 6892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 442c7e31793c467bad3a792af6421363: Sending DeleteTablet for 3 replicas of tablet 0fdd47b394144c46946a09abe0aa777f
I20260925 04:35:10.468398 7320 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdd47b394144c46946a09abe0aa777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:10 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:35348
Sep 25 04:35:10 dist-test-slave-r8r3 krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310909, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:10.469898 7042 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdd47b394144c46946a09abe0aa777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:10 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:44774
I20260925 04:35:10.470674 7184 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdd47b394144c46946a09abe0aa777f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:10 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:52002
I20260925 04:35:10.472028 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 6977
I20260925 04:35:10.475243 7434 tablet_replica.cc:330] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: stopping tablet replica
I20260925 04:35:10.475611 7434 raft_consensus.cc:2229] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:35:10.476002 7434 raft_consensus.cc:2258] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:35:10.476109 7432 tablet_replica.cc:330] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: stopping tablet replica
I20260925 04:35:10.476403 7432 raft_consensus.cc:2229] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:35:10.476701 7432 raft_consensus.cc:2258] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:35:10.478295 7432 ts_tablet_manager.cc:1905] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:35:10.481243 7432 ts_tablet_manager.cc:1918] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:35:10.481348 7432 log.cc:1196] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/0fdd47b394144c46946a09abe0aa777f
I20260925 04:35:10.481706 7432 ts_tablet_manager.cc:1939] T 0fdd47b394144c46946a09abe0aa777f P d8c105d8692a48f192be5479c0ebd5f6: Deleting consensus metadata
I20260925 04:35:10.481894 7434 ts_tablet_manager.cc:1905] T 0fdd47b394144c46946a09abe0aa777f P 57fa5227c6524fcb92339987ee18c9ba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:35:10.482980 6877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:42953 (error 108)
I20260925 04:35:10.483012 6879 catalog_manager.cc:4841] TS d8c105d8692a48f192be5479c0ebd5f6 (127.27.46.195:41991): tablet 0fdd47b394144c46946a09abe0aa777f (table test-table [id=b1ef3515e0294ee39228059735c0ff80]) successfully deleted
I20260925 04:35:10.484041 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 7119
I20260925 04:35:10.491428 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.491128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:42953 (local address 127.0.0.1:44790)
0925 04:35:10.491242 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.491327 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:42953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 04:35:10.491612 6877 catalog_manager.cc:4584] TS f4be5ad5f60c459287e62b15e6738b7f (127.27.46.193:42953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fdd47b394144c46946a09abe0aa777f: Network error: Client connection negotiation failed: client connection to 127.27.46.193:42953: connect: Connection refused (error 111)
I20260925 04:35:10.494064 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 7256
I20260925 04:35:10.496544 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:10.496165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:45237 (local address 127.0.0.1:52008)
0925 04:35:10.496383 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:10.496448 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 04:35:10.504431 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 6861
I20260925 04:35:10.513998 27835 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:35:10.858748 27835 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:35:10.858877 27835 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:35:11.203445 27835 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:35:11.203 UTC [6443] LOG: received smart shutdown request
2026-09-25 04:35:11.207 UTC [6443] LOG: background worker "logical replication launcher" (PID 6452) exited with exit code 1
2026-09-25 04:35:11.208 UTC [6447] LOG: shutting down
2026-09-25 04:35:11.380 UTC [6443] LOG: database system is shut down
2026-09-25T04:35:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47206 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:35:11 dist-test-slave-r8r3 krb5kdc[7445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:35:11 dist-test-slave-r8r3 krb5kdc[7445](info): set up 2 sockets
Sep 25 04:35:11 dist-test-slave-r8r3 krb5kdc[7445](info): commencing operation
krb5kdc: starting...
W20260925 04:35:12.552816 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:35:12 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:35:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangeradmin_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangeradmin_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangerlookup_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangerlookup_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
I20260925 04:35:12.712424 27835 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjz8mnA/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/postgres -l logfile start
2026-09-25 04:35:15.317 UTC [7475] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:35:15.317 UTC [7475] LOG: listening on IPv4 address "127.27.46.212", port 51457
2026-09-25 04:35:15.321 UTC [7475] LOG: listening on Unix socket "/tmp/.s.PGSQL.51457"
2026-09-25 04:35:15.343 UTC [7478] LOG: database system was shut down at 2026-09-25 04:35:13 UTC
2026-09-25 04:35:15.347 UTC [7475] LOG: database system is ready to accept connections
I20260925 04:35:16.342576 27835 mini_postgres.cc:96] Postgres bound to 51457
2026-09-25 04:35:16.349 UTC [7486] FATAL: database "slave" does not exist
127.27.46.212:51457 - accepting connections
I20260925 04:35:16.349639 27835 mini_ranger.cc:161] Starting Ranger...
I20260925 04:35:16.365093 27835 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:35:16.881253 27835 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:35:16.881349 27835 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin
2026-09-25 04:35:17,152 [I] DB FLAVOR :POSTGRES
2026-09-25 04:35:17,153 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:35:17,153 [I] Checking connection..
2026-09-25 04:35:17,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:35:17,408 [I] Checking connection passed.
2026-09-25 04:35:17,409 [I] --------- Verifying version history table ---------
2026-09-25 04:35:17,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:35:17,667 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:35:17,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:35:17,906 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:35:17,907 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:35:17,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:35:18.151 UTC [7551] WARNING: there is no transaction in progress
2026-09-25 04:35:18,158 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:35:18,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:35:18,423 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:35:18,423 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:35:18,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:35:18,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:35:18,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r8r3.c.gcp-upstream.internal','N') ;"
2026-09-25 04:35:19,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:35:19,365 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:35:19,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:35:19,599 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:35:19,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:35:19,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:35:20,064 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:35:20,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:35:21.048 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.064 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.082 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.098 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.114 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.133 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.192 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.198 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.213 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.223 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.234 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.249 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.260 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.267 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.275 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21.282 UTC [7660] WARNING: there is no transaction in progress
2026-09-25 04:35:21,963 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:35:21,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:35:22,212 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:35:22,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:35:22,458 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:35:22,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:35:22,713 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:35:22,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:35:22,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:35:23,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r8r3.c.gcp-upstream.internal';"
2026-09-25 04:35:23,450 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:35:23,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:35:23,695 [I] DB_PATCHES have already been applied
I20260925 04:35:23.701476 27835 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjz8mnA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:35:23.701598 27835 mini_ranger.cc:193] Using host: 127.27.46.212
I20260925 04:35:23.705197 27835 mini_ranger.cc:227] Ranger admin URL: http://127.27.46.212:34171
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:35:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:35:24.832370 27835 mini_ranger.cc:160] Time spent starting Ranger: real 8.483s user 0.000s sys 0.010s
Sep 25, 2026 4:35:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:35:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:35:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:35:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:35:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:35:45.863099 27835 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangerkms_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/rangerkms_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/HTTP_127.27.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:35:45 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310945, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:35:45.947476 27835 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:35:45.957176 27835 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:35:46.797116 27835 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:35:46.797224 27835 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms
I20260925 04:35:46.797250 27835 mini_ranger_kms.cc:227] Using postgres at 127.27.46.212:51457
2026-09-25 04:35:46,979 [I] DB FLAVOR :POSTGRES
2026-09-25 04:35:46,979 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:35:46,979 [I] Checking connection
2026-09-25 04:35:46,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:35:47,206 [I] connection success
2026-09-25 04:35:47,206 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:35:47,206 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:35:47,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:35:47,434 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:35:47,434 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:35:47,435 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:35:47,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.46.212:51457/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:35:47,675 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:35:48.107844 27835 mini_ranger_kms.cc:310] Created kms service
I20260925 04:35:48.283727 27835 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:35:48.325882 27835 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:35:48 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310945, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:48.739936 27835 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:35:48.740231 27835 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjz8mnA/build/debug/bin/postgresql.jar:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjz8mnA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:35:48.740257 27835 mini_ranger_kms.cc:240] Using host: 127.27.46.212
I20260925 04:35:48.744712 27835 mini_ranger_kms.cc:276] Ranger KMS PID: 7840
I20260925 04:35:48.744793 27835 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.46.212:44421
Sep 25, 2026 4:35:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:35:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:35:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:35:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:35:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:35:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:35:50.994561 27835 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.047s user 0.007s sys 0.159s
I20260925 04:35:50.994694 27835 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:35:50.994787 27835 mini_ranger_kms.cc:401] 127.27.46.212:44421/kms/v1/keys
Sep 25 04:35:51 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790310951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:35:51 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
Sep 25 04:35:51 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310951, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.46.212@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
Sep 25 04:35:54 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:35:54.080322 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39375
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:39375
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:54.199687 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:54.200035 7891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:54.200119 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:54.204352 7891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:35:54.204470 7891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:54.204506 7891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:54.204546 7891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:35:54.204577 7891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:35:54.210592 7891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.46.212:44421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:39375
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:54.211819 7891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:54.212915 7891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:54.222218 7896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:54.222285 7891 server_base.cc:1034] running on GCE node
W20260925 04:35:54.222206 7899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:54.222208 7897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:54.223021 7891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:54.224203 7891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:54.225391 7891 hybrid_clock.cc:648] HybridClock initialized: now 1790310954225365 us; error 41 us; skew 500 ppm
Sep 25 04:35:54 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310954, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:54.232223 7891 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:35:54.233206 7891 webserver.cc:460] Webserver started at http://127.27.46.254:44121/ using document root <none> and password file <none>
I20260925 04:35:54.233848 7891 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:54.233934 7891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:54.234179 7891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:54 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310954, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:54.290016 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "0aac5bf346474015a42a99128c69210c"
format_stamp: "Formatted at 2026-09-25 04:35:54 on dist-test-slave-r8r3"
server_key: "25170983545fe4b5c16915b8596cb327"
server_key_iv: "f33b653435bab90ae45b365a803b44d1"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:54.290799 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "0aac5bf346474015a42a99128c69210c"
format_stamp: "Formatted at 2026-09-25 04:35:54 on dist-test-slave-r8r3"
server_key: "25170983545fe4b5c16915b8596cb327"
server_key_iv: "f33b653435bab90ae45b365a803b44d1"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:54.295814 7891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:35:54.314035 7906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:54.315939 7891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:35:54.316107 7891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "0aac5bf346474015a42a99128c69210c"
format_stamp: "Formatted at 2026-09-25 04:35:54 on dist-test-slave-r8r3"
server_key: "25170983545fe4b5c16915b8596cb327"
server_key_iv: "f33b653435bab90ae45b365a803b44d1"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:54.316269 7891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:54.328737 7891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:54.331879 7891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:54.332353 7891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:54.340286 7891 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:39375
I20260925 04:35:54.340325 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:39375 every 8 connection(s)
I20260925 04:35:54.341332 7891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:35:54.344236 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:54.349897 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Bootstrap starting.
I20260925 04:35:54.350409 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 7891
I20260925 04:35:54.350539 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:35:54.350834 27835 external_mini_cluster.cc:1428] Setting key 0f3d23a97e75ce9feb433f927346990d
I20260925 04:35:54.352696 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:54.353629 7960 log.cc:824] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:54.356037 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: No bootstrap required, opened a new log
Sep 25 04:35:54 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:54.360343 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:54.360658 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:54.360736 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aac5bf346474015a42a99128c69210c, State: Initialized, Role: FOLLOWER
I20260925 04:35:54.361274 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:54.361425 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:35:54.361526 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:35:54.361639 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:54.362720 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:54.363088 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aac5bf346474015a42a99128c69210c; no voters:
I20260925 04:35:54.363514 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:35:54.363585 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:35:54.363865 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 LEADER]: Becoming Leader. State: Replica: 0aac5bf346474015a42a99128c69210c, State: Running, Role: LEADER
I20260925 04:35:54.364246 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:54.364964 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:35:54.366243 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aac5bf346474015a42a99128c69210c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } } }
I20260925 04:35:54.366367 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:54.366541 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:54.352249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.27.46.254:39375)
0925 04:35:54.352719 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:54.352741 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:54.352797 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:35:54.353692 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:54.353717 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:54.354155 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:54.354436 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:54.355225 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:54.356665 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:54.357541 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:54.358257 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:54.358420 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:54.361160 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:54.361165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:54.361179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:54.361206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:54.362911 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:54.363377 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:54.363385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:54.363396 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:54.363518 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:54.363817 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:54.363821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:54.363822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:54.364383 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:54.365070 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:54.365246 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 04:35:54.365534 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260925 04:35:54.366707 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aac5bf346474015a42a99128c69210c. Latest consensus state: current_term: 1 leader_uuid: "0aac5bf346474015a42a99128c69210c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } } }
I20260925 04:35:54.366844 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:54.367355 7960 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:35:54.367794 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:35:54.374171 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:35:54.381377 7974 catalog_manager.cc:1347] Generated new cluster ID: 94838efb3ad64d2d909cb3636877e700
I20260925 04:35:54.381572 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:35:54.405702 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:35:54.407037 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:35:54.414867 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Generated new TSK 0
I20260925 04:35:54.415704 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:35:55.066577 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:55.212044 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:55.212419 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:55.212502 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:55.217988 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:55.218050 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:55.218139 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:35:55.224277 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:55.225701 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:55.226994 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:55.238760 8013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:55.238760 8014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:55.239192 8006 server_base.cc:1034] running on GCE node
W20260925 04:35:55.239679 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:55.240113 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:55.240851 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:55.242018 8006 hybrid_clock.cc:648] HybridClock initialized: now 1790310955241988 us; error 51 us; skew 500 ppm
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:55.244812 8006 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:35:55.245793 8006 webserver.cc:460] Webserver started at http://127.27.46.193:32987/ using document root <none> and password file <none>
I20260925 04:35:55.246382 8006 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:55.246513 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:55.246799 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:55.259397 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "244242f81a844f9e95042c5fb5c5060d"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b813fa0deab1063eecb21bdb4beef08f"
server_key_iv: "f21d4df1d4ce94dfbde4c5686f656e52"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.260143 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "244242f81a844f9e95042c5fb5c5060d"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b813fa0deab1063eecb21bdb4beef08f"
server_key_iv: "f21d4df1d4ce94dfbde4c5686f656e52"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.265385 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 04:35:55.285595 8024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:55.287385 8006 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:35:55.287521 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "244242f81a844f9e95042c5fb5c5060d"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b813fa0deab1063eecb21bdb4beef08f"
server_key_iv: "f21d4df1d4ce94dfbde4c5686f656e52"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.287652 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:55.309000 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:55.313160 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:55.313352 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:55.314087 8006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:55.315172 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:55.315243 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:55.315295 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:55.315348 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 04:35:55.326169 8006 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:45373
I20260925 04:35:55.327304 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:35:55.330204 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8006
I20260925 04:35:55.330322 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:35:55.330644 27835 external_mini_cluster.cc:1428] Setting key 9239d027c09b2c14c69831f161c4daa5
I20260925 04:35:55.329463 8141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:45373 every 8 connection(s)
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:35:55.360930 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:55.337731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:42989 (local address 127.27.46.254:39375)
0925 04:35:55.342059 (+ 4328us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:55.342068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:55.342084 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:35:55.342127 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:55.342131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:55.342182 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:55.342268 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:55.343857 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.344730 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.346485 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.346829 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.346925 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:55.354060 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:55.354065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:55.354067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:55.354098 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:55.356126 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:55.358605 (+ 2479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:55.358609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:55.358611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:55.358714 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:55.359201 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:55.359205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:55.359207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:55.359651 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:55.359777 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:55.360504 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 04:35:55.360723 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4238,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:35:55.361929 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:55.337022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.193:42989)
0925 04:35:55.340374 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:55.340453 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:55.341599 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:55.342601 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:55.342611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:55.342977 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:55.343679 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:55.343701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.344884 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.344888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:55.346333 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:55.346344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.347104 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.347107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:55.347293 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:55.351820 (+ 4527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:55.351843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:55.353772 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:55.356277 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:55.356284 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:55.356301 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:55.356665 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:55.358830 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:55.358834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:55.358837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:55.359042 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:55.359797 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:55.359805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:55.360162 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 04:35:55.360394 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 04:35:55.360767 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3103,"thread_start_us":161,"threads_started":1}
I20260925 04:35:55.363685 8142 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:55.363945 8142 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:55.364562 8142 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:55.366381 7923 ts_manager.cc:194] Registered new tserver with Master: 244242f81a844f9e95042c5fb5c5060d (127.27.46.193:45373)
I20260925 04:35:55.368206 7923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:42989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:35:55.380389 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:35:55.507695 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:55.507972 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:55.508073 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:55.513031 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:55.513116 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:55.513216 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:35:55.519227 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:55.520464 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:55.521689 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:55.531378 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:55.531390 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:55.531510 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:55.532114 8148 server_base.cc:1034] running on GCE node
I20260925 04:35:55.532533 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:55.533228 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:55.534401 8148 hybrid_clock.cc:648] HybridClock initialized: now 1790310955534363 us; error 55 us; skew 500 ppm
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:55.537204 8148 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:35:55.538205 8148 webserver.cc:460] Webserver started at http://127.27.46.194:36499/ using document root <none> and password file <none>
I20260925 04:35:55.538746 8148 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:55.538844 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:55.539057 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:55.552167 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "c0a814c779b046789ec945fd09bbd263"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "a8ce7b39de3b6d07b31f96035e7fad81"
server_key_iv: "e3a9c784c0482bb807f23014d7665df7"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.552955 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "c0a814c779b046789ec945fd09bbd263"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "a8ce7b39de3b6d07b31f96035e7fad81"
server_key_iv: "e3a9c784c0482bb807f23014d7665df7"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.558310 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:35:55.577430 8164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:55.579213 8148 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260925 04:35:55.579355 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c0a814c779b046789ec945fd09bbd263"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "a8ce7b39de3b6d07b31f96035e7fad81"
server_key_iv: "e3a9c784c0482bb807f23014d7665df7"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.579484 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:55.600422 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:55.606211 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:55.606470 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:55.607204 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:55.608528 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:55.608630 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:55.608721 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:55.608779 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:55.619823 8148 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:35211
I20260925 04:35:55.621532 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:35:55.627871 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8148
I20260925 04:35:55.627980 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:35:55.628309 27835 external_mini_cluster.cc:1428] Setting key 82e45113f411472d9935bc29745587ab
I20260925 04:35:55.633975 8279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:35211 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:55.674477 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:55.636811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.194:43329)
0925 04:35:55.646834 (+ 10023us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:55.646951 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:55.648201 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:55.649112 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:55.649122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:55.649503 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:55.650273 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:55.650293 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.652107 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.652111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:55.655785 (+ 3674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:55.655797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.656647 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.656651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:55.656859 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:55.662032 (+ 5173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:55.662057 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:55.663923 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:55.670702 (+ 6779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:55.670710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:55.670730 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:55.671146 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:55.671501 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:55.671504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:55.671506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:55.671719 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:55.672302 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:55.672309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:55.672669 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 04:35:55.672918 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 04:35:55.673295 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9702,"thread_start_us":134,"threads_started":1}
I20260925 04:35:55.675987 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:55.636261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:43329 (local address 127.27.46.254:39375)
0925 04:35:55.636397 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:55.636401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:55.647088 (+ 10687us) server_negotiation.cc:407] Connection header received
0925 04:35:55.648495 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:55.648499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:55.648558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:55.648614 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:55.650885 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.651719 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.655951 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:55.656319 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:55.656412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:55.664165 (+ 7753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:55.664169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:55.664172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:55.664204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:55.670566 (+ 6362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:55.671281 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:55.671285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:55.671287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:55.671382 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:55.671842 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:55.671845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:55.671847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:55.672141 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:55.672242 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:55.675639 (+ 3397us) server_negotiation.cc:299] Negotiation successful
0925 04:35:55.675785 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:35:55.682118 8280 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:55.682396 8280 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:55.683046 8280 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:55.684304 7923 ts_manager.cc:194] Registered new tserver with Master: c0a814c779b046789ec945fd09bbd263 (127.27.46.194:35211)
I20260925 04:35:55.685122 7923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:43329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:35:55.701069 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:55.882562 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:55.883064 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:55.883204 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:55.889398 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:55.889593 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:55.889744 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:35:55.897845 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:55.899587 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:55.901052 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:55.915261 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:55.915627 8285 server_base.cc:1034] running on GCE node
W20260925 04:35:55.916204 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:55.916550 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:55.917171 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:55.918083 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:55.919386 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790310955919320 us; error 80 us; skew 500 ppm
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:55.923532 8285 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:35:55.925087 8285 webserver.cc:460] Webserver started at http://127.27.46.195:34089/ using document root <none> and password file <none>
I20260925 04:35:55.925956 8285 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:55.926095 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:35:55.926364 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:55.949091 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "f43abcd0252344a7b082a16932c902a8"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b2b6581e58a0f6e862cdeaa8edd31050"
server_key_iv: "d24126e236cb634df4ccce0d43f824de"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:55.949998 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "f43abcd0252344a7b082a16932c902a8"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b2b6581e58a0f6e862cdeaa8edd31050"
server_key_iv: "d24126e236cb634df4ccce0d43f824de"
server_key_version: "kuduclusterkey@0"
Sep 25 04:35:55 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:55.979228 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.004s sys 0.000s
I20260925 04:35:56.000159 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.001942 8285 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 04:35:56.002084 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "f43abcd0252344a7b082a16932c902a8"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b2b6581e58a0f6e862cdeaa8edd31050"
server_key_iv: "d24126e236cb634df4ccce0d43f824de"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:56.002224 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:56.022104 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:56.026211 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:56.026409 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:56.027064 8285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:56.028036 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:56.028100 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.028146 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:56.028165 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.054114 8285 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:46025
I20260925 04:35:56.055275 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:35:56.062101 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:46025 every 8 connection(s)
I20260925 04:35:56.064872 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8285
I20260925 04:35:56.064986 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:35:56.065312 27835 external_mini_cluster.cc:1428] Setting key 989c7234728adcc248e7c082c7f93a7a
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:56.096541 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.064126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:47199 (local address 127.27.46.254:39375)
0925 04:35:56.064263 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:56.064267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:56.069526 (+ 5259us) server_negotiation.cc:407] Connection header received
0925 04:35:56.073635 (+ 4109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:56.073639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:56.073702 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:56.073762 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:56.077016 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.077891 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.081596 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.081999 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.082081 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.091601 (+ 9520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:56.091605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:56.091608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:56.091638 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:56.093426 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.094219 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.094222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.094224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.094308 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.094949 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.094953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.094955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.095248 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:56.095352 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:56.096180 (+ 828us) server_negotiation.cc:299] Negotiation successful
0925 04:35:56.096343 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:35:56.098286 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.064772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.195:47199)
0925 04:35:56.069334 (+ 4562us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:56.069404 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:56.071665 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:56.074230 (+ 2565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:56.074241 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:56.074621 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:56.075321 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.075342 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.078061 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.078064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.079507 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.079518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.082296 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.082300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.082589 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.083490 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:56.083531 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:56.091380 (+ 7849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:56.093606 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.093613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.093635 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.094102 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.094572 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.094575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.094577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.094807 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.095437 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:56.095445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:56.095824 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 04:35:56.096732 (+ 908us) client_negotiation.cc:240] Negotiation successful
0925 04:35:56.097117 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4293,"thread_start_us":145,"threads_started":1}
I20260925 04:35:56.099510 8416 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:56.099767 8416 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.100302 8416 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:56.101842 7922 ts_manager.cc:194] Registered new tserver with Master: f43abcd0252344a7b082a16932c902a8 (127.27.46.195:46025)
I20260925 04:35:56.102568 7922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:47199
I20260925 04:35:56.103798 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:35:56.104672 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8006
I20260925 04:35:56.124182 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8148
I20260925 04:35:56.141669 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8285
I20260925 04:35:56.151887 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 7891
I20260925 04:35:56.160585 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39375
--webserver_interface=127.27.46.254
--webserver_port=44121
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:39375
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:35:56.289569 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:56.289853 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:56.289893 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:56.293861 8419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:35:56.293906 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:56.293924 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:56.293939 8419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:35:56.293954 8419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:35:56.299274 8419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:39375
--ranger_config_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:56.300323 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:56.301313 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:56.309842 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:56.309850 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:56.309886 8419 server_base.cc:1034] running on GCE node
W20260925 04:35:56.309850 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:56.310616 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:56.311659 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:56.312810 8419 hybrid_clock.cc:648] HybridClock initialized: now 1790310956312788 us; error 31 us; skew 500 ppm
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:56.315668 8419 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:35:56.316558 8419 webserver.cc:460] Webserver started at http://127.27.46.254:44121/ using document root <none> and password file <none>
I20260925 04:35:56.317087 8419 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:56.317178 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:56.365216 8419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:35:56.367578 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.368939 8419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:35:56.369091 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "0aac5bf346474015a42a99128c69210c"
format_stamp: "Formatted at 2026-09-25 04:35:54 on dist-test-slave-r8r3"
server_key: "25170983545fe4b5c16915b8596cb327"
server_key_iv: "f33b653435bab90ae45b365a803b44d1"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:56.369740 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:56.383141 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:56.385743 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:56.385918 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:56.392797 8419 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:39375
I20260925 04:35:56.392827 8488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:39375 every 8 connection(s)
I20260925 04:35:56.393891 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:35:56.398118 8489 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:35:56.398638 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8419
I20260925 04:35:56.399053 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:45373
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:39375
--webserver_port=32987
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:35:56.399806 8489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Bootstrap starting.
I20260925 04:35:56.416323 8489 log.cc:824] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:56.423136 8489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:35:56.423617 8489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Bootstrap complete.
I20260925 04:35:56.428087 8489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:56.428519 8489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aac5bf346474015a42a99128c69210c, State: Initialized, Role: FOLLOWER
I20260925 04:35:56.429067 8489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:56.429227 8489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:35:56.429312 8489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:35:56.429435 8489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:35:56.431365 8489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:56.431702 8489 leader_election.cc:302] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aac5bf346474015a42a99128c69210c; no voters:
I20260925 04:35:56.432020 8489 leader_election.cc:288] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:35:56.432160 8493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:35:56.432394 8493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [term 2 LEADER]: Becoming Leader. State: Replica: 0aac5bf346474015a42a99128c69210c, State: Running, Role: LEADER
I20260925 04:35:56.432693 8493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } }
I20260925 04:35:56.433275 8489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:35:56.434182 8494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0aac5bf346474015a42a99128c69210c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } } }
I20260925 04:35:56.434316 8494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:56.434545 8494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aac5bf346474015a42a99128c69210c. Latest consensus state: current_term: 2 leader_uuid: "0aac5bf346474015a42a99128c69210c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aac5bf346474015a42a99128c69210c" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39375 } } }
I20260925 04:35:56.434626 8494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c [sys.catalog]: This master's current role is: LEADER
I20260925 04:35:56.435317 8502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:35:56.435619 8489 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:35:56.441408 8502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:35:56.442286 8502 catalog_manager.cc:1259] Loaded cluster ID: 94838efb3ad64d2d909cb3636877e700
I20260925 04:35:56.442342 8502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:35:56.445062 8502 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:35:56.445974 8502 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Loaded TSK: 0
I20260925 04:35:56.447188 8502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:35:56.537849 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:56.538163 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:56.538242 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:56.543746 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:56.543843 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:56.543951 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:35:56.549989 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:45373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=32987
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:56.551106 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:56.552129 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:56.562947 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:56.562950 8529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:56.563306 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:56.563966 8491 server_base.cc:1034] running on GCE node
I20260925 04:35:56.564436 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:56.565169 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:56.566330 8491 hybrid_clock.cc:648] HybridClock initialized: now 1790310956566300 us; error 52 us; skew 500 ppm
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:56.569886 8491 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:35:56.570967 8491 webserver.cc:460] Webserver started at http://127.27.46.193:32987/ using document root <none> and password file <none>
I20260925 04:35:56.571712 8491 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:56.571784 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:56.601630 8491 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.001s
I20260925 04:35:56.606621 8539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.608103 8491 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 04:35:56.608259 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "244242f81a844f9e95042c5fb5c5060d"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b813fa0deab1063eecb21bdb4beef08f"
server_key_iv: "f21d4df1d4ce94dfbde4c5686f656e52"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:56.608907 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:56.630492 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:56.634245 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:56.634447 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:56.635212 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:56.636312 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:56.636380 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.636456 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:56.636482 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.648046 8491 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:45373
I20260925 04:35:56.648073 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:45373 every 8 connection(s)
I20260925 04:35:56.649098 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:35:56.656996 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8491
I20260925 04:35:56.657456 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:35211
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:39375
--webserver_port=36499
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:56.671430 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.651647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.193:54215)
0925 04:35:56.652423 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:56.652478 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:56.653294 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:56.654496 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:56.654504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:56.654834 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:56.655286 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.655300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.656590 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.656594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.657899 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.657914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.661620 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.661624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.661814 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.662534 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:56.662555 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:56.664313 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:56.666636 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.666643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.666660 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.667024 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.667466 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.667471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.667476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.667764 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.668779 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:56.668787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:56.669244 (+ 457us) client_negotiation.cc:769] Sending connection context
0925 04:35:56.669665 (+ 421us) client_negotiation.cc:240] Negotiation successful
0925 04:35:56.670208 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":142,"threads_started":1}
I20260925 04:35:56.673167 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.651467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:54215 (local address 127.27.46.254:39375)
0925 04:35:56.652141 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:56.652165 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:56.652593 (+ 428us) server_negotiation.cc:407] Connection header received
0925 04:35:56.653786 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:56.653798 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:56.654093 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:56.654411 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:56.655471 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.656453 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.660386 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.661165 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.661370 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.664552 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:56.664562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:56.664583 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:56.664642 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:56.666500 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.667169 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.667179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.667187 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.667322 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.667934 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.667940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.667970 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.668560 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:56.668721 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:56.672066 (+ 3345us) server_negotiation.cc:299] Negotiation successful
0925 04:35:56.672338 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":110,"threads_started":1}
I20260925 04:35:56.675972 8654 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:56.676204 8654 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.676781 8654 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
W20260925 04:35:56.679323 8654 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:39375 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:35:56.679459 8654 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.679657 8654 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:56.680799 8654 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.681035 8654 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
W20260925 04:35:56.681977 8654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:35:56.814519 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:56.814956 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:56.815026 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:56.820175 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:56.820236 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:56.820330 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:35:56.826177 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:35211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=36499
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:56.827445 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:56.828584 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:56.842128 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:56.842118 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:56.843564 8658 server_base.cc:1034] running on GCE node
W20260925 04:35:56.845321 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:56.845881 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:56.846689 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:56.854094 8658 hybrid_clock.cc:648] HybridClock initialized: now 1790310956854045 us; error 58 us; skew 500 ppm
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:56.857151 8658 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:35:56.858136 8658 webserver.cc:460] Webserver started at http://127.27.46.194:36499/ using document root <none> and password file <none>
I20260925 04:35:56.858678 8658 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:56.858742 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:56.881825 8658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:35:56.897694 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.899530 8658 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260925 04:35:56.899705 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c0a814c779b046789ec945fd09bbd263"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "a8ce7b39de3b6d07b31f96035e7fad81"
server_key_iv: "e3a9c784c0482bb807f23014d7665df7"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:56.900648 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:56.935987 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:56.939711 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:56.939957 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:56.940766 8658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:56.941927 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:56.941985 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.942039 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:56.942065 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:56.954587 8658 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:35211
I20260925 04:35:56.954731 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:35211 every 8 connection(s)
I20260925 04:35:56.955688 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:35:56.958529 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8658
I20260925 04:35:56.958989 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.46.212:44421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:46025
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:39375
--webserver_port=34089
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:35:56 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:56.972832 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.957529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:60617 (local address 127.27.46.254:39375)
0925 04:35:56.957669 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:56.957673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:56.959517 (+ 1844us) server_negotiation.cc:407] Connection header received
0925 04:35:56.960912 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:56.960916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:56.960984 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:56.961064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:56.962592 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.963339 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.964495 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.964807 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.964919 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.967926 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:56.967933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:56.967937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:56.967973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:56.969841 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.970564 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.970570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.970574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.970679 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:56.971174 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:56.971180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:56.971183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:56.971446 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:56.971542 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:56.972428 (+ 886us) server_negotiation.cc:299] Negotiation successful
0925 04:35:56.972610 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:35:56.974014 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:56.958075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.194:60617)
0925 04:35:56.959336 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:56.959397 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:56.960687 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:56.961253 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:56.961261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:56.961666 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:56.962397 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.962420 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.963499 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.963501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.964365 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:56.964374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:56.964979 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:56.964983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:56.965220 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:56.965852 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:56.965877 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:56.967694 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:56.970004 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.970012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.970030 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.970428 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.970809 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:56.970814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:56.970817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:56.971050 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:56.971638 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:56.971646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:56.972031 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 04:35:56.972304 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 04:35:56.972714 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":152,"threads_started":1}
I20260925 04:35:56.975330 8790 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:56.975579 8790 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.976166 8790 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
W20260925 04:35:56.977916 8790 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:39375 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:35:56.978031 8790 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.978188 8790 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:56.978951 8790 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:56.979107 8790 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
W20260925 04:35:56.979779 8790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:35:57.101116 8793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:35:57.101481 8793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:35:57.101604 8793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:35:57.108106 8793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:35:57.108240 8793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:35:57.108364 8793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:35:57.115180 8793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.46.212:44421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:46025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=34089
--tserver_master_addrs=127.27.46.254:39375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:35:57.116535 8793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:35:57.117720 8793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:35:57.129738 8798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:57.129762 8793 server_base.cc:1034] running on GCE node
W20260925 04:35:57.129782 8799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:35:57.130056 8801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:35:57.130594 8793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:35:57.131392 8793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:35:57.134660 8793 hybrid_clock.cc:648] HybridClock initialized: now 1790310957134617 us; error 52 us; skew 500 ppm
Sep 25 04:35:57 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:57.138224 8793 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:35:57.139322 8793 webserver.cc:460] Webserver started at http://127.27.46.195:34089/ using document root <none> and password file <none>
I20260925 04:35:57.139976 8793 fs_manager.cc:359] Metadata directory not provided
I20260925 04:35:57.140053 8793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:35:57 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:57.164182 8793 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:35:57.167125 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:57.168490 8793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:35:57.168645 8793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "f43abcd0252344a7b082a16932c902a8"
format_stamp: "Formatted at 2026-09-25 04:35:55 on dist-test-slave-r8r3"
server_key: "b2b6581e58a0f6e862cdeaa8edd31050"
server_key_iv: "d24126e236cb634df4ccce0d43f824de"
server_key_version: "kuduclusterkey@0"
I20260925 04:35:57.169201 8793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:35:57.186647 8793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:35:57.190096 8793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:35:57.190303 8793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:35:57.190989 8793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:35:57.191932 8793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:35:57.192003 8793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:57.192111 8793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:35:57.192152 8793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:35:57.202669 8793 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:46025
I20260925 04:35:57.202976 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:46025 every 8 connection(s)
I20260925 04:35:57.203763 8793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:35:57.210969 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 8793
Sep 25 04:35:57 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:35:57.219506 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:57.205792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:53229 (local address 127.27.46.254:39375)
0925 04:35:57.205913 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:57.205917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:57.206995 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 04:35:57.208222 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:57.208225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:57.208272 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:57.208346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:57.209838 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.210351 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.212336 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.212572 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.212683 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:57.215353 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:57.215357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:57.215360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:57.215389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:57.217224 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.217881 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.217885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.217887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.217997 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.218342 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.218346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.218347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.218547 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:57.218671 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:57.219188 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 04:35:57.219315 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":34}
I20260925 04:35:57.220553 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:57.206262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39375 (local address 127.27.46.195:53229)
0925 04:35:57.206814 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:57.206871 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:57.207987 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:57.208568 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:57.208579 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:57.208960 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:35:57.209663 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:57.209683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.210481 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.210485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:57.212190 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:57.212202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.212741 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.212745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:57.212976 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:57.213729 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:35:57.213753 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:35:57.215151 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:35:57.217360 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:57.217366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:57.217382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:57.217771 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:57.218092 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:35:57.218094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:35:57.218096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:35:57.218230 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:35:57.218668 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:35:57.218673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:35:57.218941 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 04:35:57.219126 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:35:57.219435 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260925 04:35:57.221904 8935 heartbeater.cc:347] Connected to a master server at 127.27.46.254:39375
I20260925 04:35:57.222173 8935 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:57.222781 8935 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:57.223798 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:57.211617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.27.46.254:39375)
0925 04:35:57.212492 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:57.212500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:57.212514 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:35:57.212552 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:57.212555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:57.212602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:57.212686 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:57.213705 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.214511 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.216059 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.216338 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.216456 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:57.219694 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:57.219698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:57.219700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:57.219723 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:57.221636 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.222162 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.222165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.222167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.222258 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.222564 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.222567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.222569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.222773 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:57.222992 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:57.223251 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:35:57.223645 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":791,"thread_start_us":140,"threads_started":1}
I20260925 04:35:57.224273 8453 ts_manager.cc:194] Registered new tserver with Master: f43abcd0252344a7b082a16932c902a8 (127.27.46.195:46025)
I20260925 04:35:57.225564 8453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:53229
Sep 25 04:35:57 dist-test-slave-r8r3 krb5kdc[7445](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:35:57.682196 8654 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:57.682509 8654 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:57.683274 8453 ts_manager.cc:194] Registered new tserver with Master: 244242f81a844f9e95042c5fb5c5060d (127.27.46.193:45373)
I20260925 04:35:57.684098 8453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:54215
Sep 25 04:35:57 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:57.980038 8790 heartbeater.cc:464] Registering TS with master...
I20260925 04:35:57.980346 8790 heartbeater.cc:511] Master 127.27.46.254:39375 requested a full tablet report, sending...
I20260925 04:35:57.981185 8453 ts_manager.cc:194] Registered new tserver with Master: c0a814c779b046789ec945fd09bbd263 (127.27.46.194:35211)
I20260925 04:35:57.982015 8453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:60617
I20260925 04:35:57.986305 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:35:57.998399 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:57.989122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.27.46.254:39375)
0925 04:35:57.989822 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:57.989829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:57.989887 (+ 58us) server_negotiation.cc:407] Connection header received
0925 04:35:57.990191 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:57.990195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:57.990257 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:57.990372 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:35:57.991221 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.991757 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.992608 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:57.992914 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:57.993015 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:57.995015 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:35:57.995020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:35:57.995027 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:35:57.995057 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:35:57.996554 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.997030 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.997039 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.997040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.997116 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:35:57.997399 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:35:57.997403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:35:57.997405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:35:57.997656 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:35:57.997798 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:57.998045 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 04:35:57.998178 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":96,"threads_started":1}
I20260925 04:35:58.002151 8453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:35:58.005648 8453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:35:58.028852 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.016780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35211 (local address 127.0.0.1:49114)
0925 04:35:58.021776 (+ 4996us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.021788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.021876 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:58.022806 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:58.022809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:58.022826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:58.022914 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.022920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.025024 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.025029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.026220 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.026227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.027619 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.027623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.028418 (+ 795us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.028467 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 04:35:58.028569 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:35:58.028691 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4935,"thread_start_us":109,"threads_started":1}
I20260925 04:35:58.029563 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.017816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.27.46.194:35211)
0925 04:35:58.018129 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.018136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.022165 (+ 4029us) server_negotiation.cc:407] Connection header received
0925 04:35:58.022253 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.022261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.022500 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.022644 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:58.023052 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.024897 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.026341 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.027265 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.027379 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.027564 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.029241 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.029359 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":141,"threads_started":1}
I20260925 04:35:58.031425 8956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.017141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:46025 (local address 127.0.0.1:40624)
0925 04:35:58.025225 (+ 8084us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.025239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.025330 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:58.026017 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:58.026020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:58.026036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:58.026111 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.026116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.027461 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.027464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.028472 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.028480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.031039 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.031043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.031132 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.031154 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:35:58.031224 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:35:58.031287 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8021,"thread_start_us":98,"threads_started":1}
I20260925 04:35:58.031868 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.017319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.27.46.195:46025)
0925 04:35:58.021621 (+ 4302us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.021627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.025431 (+ 3804us) server_negotiation.cc:407] Connection header received
0925 04:35:58.025555 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.025561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.025749 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.025872 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:58.026231 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.027342 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.029810 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.030725 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.030830 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.030977 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.031643 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.031739 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4160,"thread_start_us":104,"threads_started":1}
I20260925 04:35:58.032959 8717 tablet_service.cc:1467] Processing CreateTablet for tablet c07f50fb52904898856af61aab2f4abe (DEFAULT_TABLE table=test-table [id=abbcf8d75cb14398a394ace75292ac42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:58.034273 8717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f50fb52904898856af61aab2f4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:58.034874 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.016262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:45373 (local address 127.0.0.1:55302)
0925 04:35:58.021100 (+ 4838us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.021154 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.021293 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:58.022021 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:58.022026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:58.022046 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:58.022177 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.022184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.023647 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.023650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.024999 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.025010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.034518 (+ 9508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.034521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.034610 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.034630 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:35:58.034695 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 04:35:58.034760 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4648,"thread_start_us":186,"threads_started":1}
I20260925 04:35:58.034845 8867 tablet_service.cc:1467] Processing CreateTablet for tablet c07f50fb52904898856af61aab2f4abe (DEFAULT_TABLE table=test-table [id=abbcf8d75cb14398a394ace75292ac42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:58.035449 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.017218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.27.46.193:45373)
0925 04:35:58.017571 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.017579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.021466 (+ 3887us) server_negotiation.cc:407] Connection header received
0925 04:35:58.021598 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.021604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.021844 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.022008 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:58.022306 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.023508 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.028624 (+ 5116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.034062 (+ 5438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.034204 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.034459 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.035152 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.035255 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
I20260925 04:35:58.035919 8867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f50fb52904898856af61aab2f4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:58.038511 8584 tablet_service.cc:1467] Processing CreateTablet for tablet c07f50fb52904898856af61aab2f4abe (DEFAULT_TABLE table=test-table [id=abbcf8d75cb14398a394ace75292ac42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:35:58.039510 8584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07f50fb52904898856af61aab2f4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:35:58.047318 8963 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Bootstrap starting.
I20260925 04:35:58.047853 8962 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Bootstrap starting.
I20260925 04:35:58.050045 8962 tablet_bootstrap.cc:654] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:58.050050 8963 tablet_bootstrap.cc:654] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:58.050860 8962 log.cc:824] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:58.050860 8963 log.cc:824] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:58.051580 8961 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Bootstrap starting.
I20260925 04:35:58.053632 8961 tablet_bootstrap.cc:654] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Neither blocks nor log segments found. Creating new log.
I20260925 04:35:58.054221 8963 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: No bootstrap required, opened a new log
I20260925 04:35:58.054364 8961 log.cc:824] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Log is configured to *not* fsync() on all Append() calls
I20260925 04:35:58.054414 8963 ts_tablet_manager.cc:1397] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 04:35:58.056028 8962 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: No bootstrap required, opened a new log
I20260925 04:35:58.056210 8962 ts_tablet_manager.cc:1397] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 04:35:58.059150 8963 raft_consensus.cc:348] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.060271 8962 raft_consensus.cc:348] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.060477 8963 raft_consensus.cc:374] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:58.060532 8962 raft_consensus.cc:374] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:58.060544 8963 raft_consensus.cc:729] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244242f81a844f9e95042c5fb5c5060d, State: Initialized, Role: FOLLOWER
I20260925 04:35:58.060601 8962 raft_consensus.cc:729] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43abcd0252344a7b082a16932c902a8, State: Initialized, Role: FOLLOWER
I20260925 04:35:58.061093 8962 consensus_queue.cc:260] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.061071 8963 consensus_queue.cc:260] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.062103 8962 ts_tablet_manager.cc:1428] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:35:58.062168 8963 ts_tablet_manager.cc:1428] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:35:58.062453 8935 heartbeater.cc:503] Master 127.27.46.254:39375 was elected leader, sending a full tablet report...
I20260925 04:35:58.062481 8654 heartbeater.cc:503] Master 127.27.46.254:39375 was elected leader, sending a full tablet report...
I20260925 04:35:58.065472 8961 tablet_bootstrap.cc:492] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: No bootstrap required, opened a new log
I20260925 04:35:58.065690 8961 ts_tablet_manager.cc:1397] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20260925 04:35:58.070268 8961 raft_consensus.cc:348] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.070552 8961 raft_consensus.cc:374] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:35:58.070616 8961 raft_consensus.cc:729] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a814c779b046789ec945fd09bbd263, State: Initialized, Role: FOLLOWER
I20260925 04:35:58.071151 8961 consensus_queue.cc:260] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.072230 8961 ts_tablet_manager.cc:1428] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260925 04:35:58.072679 8790 heartbeater.cc:503] Master 127.27.46.254:39375 was elected leader, sending a full tablet report...
W20260925 04:35:58.151659 8655 tablet.cc:2369] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:35:58.207042 8936 tablet.cc:2369] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:35:58.211859 8791 tablet.cc:2369] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:35:58 dist-test-slave-r8r3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for HTTP/127.27.46.212@KRBTEST.COM
I20260925 04:35:58.241182 8968 raft_consensus.cc:482] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:35:58.241439 8968 raft_consensus.cc:504] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.243021 8968 leader_election.cc:288] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0a814c779b046789ec945fd09bbd263 (127.27.46.194:35211), f43abcd0252344a7b082a16932c902a8 (127.27.46.195:46025)
I20260925 04:35:58.250370 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.244496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:36191 (local address 127.27.46.195:46025)
0925 04:35:58.244612 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.244617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.245155 (+ 538us) server_negotiation.cc:407] Connection header received
0925 04:35:58.245352 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.245355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.245410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.245463 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:58.245942 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.246680 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.248344 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.249061 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.249215 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.249285 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.250110 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.250212 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:35:58.251030 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.244345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:50001 (local address 127.27.46.194:35211)
0925 04:35:58.244499 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.244505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.244786 (+ 281us) server_negotiation.cc:407] Connection header received
0925 04:35:58.244947 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.244951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.245003 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.245084 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:35:58.245689 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.246460 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.249521 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.250159 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.250264 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.250316 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.250801 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.250894 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:35:58.251508 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.243420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35211 (local address 127.27.46.193:50001)
0925 04:35:58.244700 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.244716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.244844 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:58.245205 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:58.245209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:58.245229 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:58.245547 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.245555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.246631 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.246634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.249329 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.249340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.250419 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.250423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.250638 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.250664 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:35:58.251253 (+ 589us) client_negotiation.cc:240] Negotiation successful
0925 04:35:58.251339 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":1331,"thread_start_us":132,"threads_started":1}
I20260925 04:35:58.252233 8971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.243870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:46025 (local address 127.27.46.193:36191)
0925 04:35:58.245074 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.245087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.245224 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:35:58.245634 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:35:58.245637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:35:58.245655 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:35:58.245786 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.245793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.246836 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.246839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.248208 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:35:58.248220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.249848 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.249851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:35:58.249954 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.249980 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:35:58.251987 (+ 2007us) client_negotiation.cc:240] Negotiation successful
0925 04:35:58.252071 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260925 04:35:58.252242 8738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f50fb52904898856af61aab2f4abe" candidate_uuid: "244242f81a844f9e95042c5fb5c5060d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a814c779b046789ec945fd09bbd263" is_pre_election: true
I20260925 04:35:58.252609 8738 raft_consensus.cc:2454] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244242f81a844f9e95042c5fb5c5060d in term 0.
I20260925 04:35:58.253525 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f50fb52904898856af61aab2f4abe" candidate_uuid: "244242f81a844f9e95042c5fb5c5060d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43abcd0252344a7b082a16932c902a8" is_pre_election: true
I20260925 04:35:58.253532 8541 leader_election.cc:302] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244242f81a844f9e95042c5fb5c5060d, c0a814c779b046789ec945fd09bbd263; no voters:
I20260925 04:35:58.253825 8872 raft_consensus.cc:2454] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244242f81a844f9e95042c5fb5c5060d in term 0.
I20260925 04:35:58.253870 8968 raft_consensus.cc:2781] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:35:58.253960 8968 raft_consensus.cc:482] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:35:58.254001 8968 raft_consensus.cc:3037] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:58.255442 8968 raft_consensus.cc:504] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.255856 8968 leader_election.cc:288] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [CANDIDATE]: Term 1 election: Requested vote from peers c0a814c779b046789ec945fd09bbd263 (127.27.46.194:35211), f43abcd0252344a7b082a16932c902a8 (127.27.46.195:46025)
I20260925 04:35:58.256330 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f50fb52904898856af61aab2f4abe" candidate_uuid: "244242f81a844f9e95042c5fb5c5060d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43abcd0252344a7b082a16932c902a8"
I20260925 04:35:58.256587 8872 raft_consensus.cc:3037] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:58.258039 8872 raft_consensus.cc:2454] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244242f81a844f9e95042c5fb5c5060d in term 1.
I20260925 04:35:58.258488 8738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07f50fb52904898856af61aab2f4abe" candidate_uuid: "244242f81a844f9e95042c5fb5c5060d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a814c779b046789ec945fd09bbd263"
I20260925 04:35:58.258621 8738 raft_consensus.cc:3037] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:35:58.258864 8541 leader_election.cc:302] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244242f81a844f9e95042c5fb5c5060d, f43abcd0252344a7b082a16932c902a8; no voters:
I20260925 04:35:58.259066 8968 raft_consensus.cc:2781] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:35:58.259325 8968 raft_consensus.cc:686] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [term 1 LEADER]: Becoming Leader. State: Replica: 244242f81a844f9e95042c5fb5c5060d, State: Running, Role: LEADER
I20260925 04:35:58.259689 8968 consensus_queue.cc:237] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } }
I20260925 04:35:58.260078 8738 raft_consensus.cc:2454] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244242f81a844f9e95042c5fb5c5060d in term 1.
I20260925 04:35:58.263659 8452 catalog_manager.cc:5495] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d reported cstate change: term changed from 0 to 1, leader changed from <none> to 244242f81a844f9e95042c5fb5c5060d (127.27.46.193). New cstate: current_term: 1 leader_uuid: "244242f81a844f9e95042c5fb5c5060d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244242f81a844f9e95042c5fb5c5060d" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 } health_report { overall_health: UNKNOWN } } }
I20260925 04:35:58.338802 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.329853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.27.46.193:45373)
0925 04:35:58.330025 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 04:35:58.330031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:35:58.330396 (+ 365us) server_negotiation.cc:407] Connection header received
0925 04:35:58.330505 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:35:58.330509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:35:58.330587 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:35:58.330713 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:35:58.331025 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.331840 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.336124 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:35:58.336524 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:35:58.336634 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:35:58.337173 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:35:58.337824 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:35:58.338050 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 04:35:58.338546 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 04:35:58.338619 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:35:58.346081 8738 raft_consensus.cc:1261] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 1 FOLLOWER]: Refusing update from remote peer 244242f81a844f9e95042c5fb5c5060d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:35:58.346084 8872 raft_consensus.cc:1261] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 1 FOLLOWER]: Refusing update from remote peer 244242f81a844f9e95042c5fb5c5060d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:35:58.346943 8973 consensus_queue.cc:1035] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f43abcd0252344a7b082a16932c902a8" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:35:58.347394 8973 consensus_queue.cc:1035] T c07f50fb52904898856af61aab2f4abe P 244242f81a844f9e95042c5fb5c5060d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a814c779b046789ec945fd09bbd263" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:35:58.359365 8977 mvcc.cc:205] Tried to move back new op lower bound from 7333113685377417216 to 7333113685035257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333113685377417216}]
I20260925 04:35:58.369817 8978 mvcc.cc:205] Tried to move back new op lower bound from 7333113685377417216 to 7333113685035257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333113685377417216}]
I20260925 04:35:58.370118 8453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:35:58.370296 8453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:35:58.372968 8453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0aac5bf346474015a42a99128c69210c: Sending DeleteTablet for 3 replicas of tablet c07f50fb52904898856af61aab2f4abe
I20260925 04:35:58.373459 8979 mvcc.cc:205] Tried to move back new op lower bound from 7333113685377417216 to 7333113685035257856. Current Snapshot: MvccSnapshot[applied={T|T < 7333113685377417216}]
I20260925 04:35:58.374013 8584 tablet_service.cc:1514] Processing DeleteTablet for tablet c07f50fb52904898856af61aab2f4abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:58 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:55302
I20260925 04:35:58.374605 8717 tablet_service.cc:1514] Processing DeleteTablet for tablet c07f50fb52904898856af61aab2f4abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:58 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:49114
I20260925 04:35:58.375061 8867 tablet_service.cc:1514] Processing DeleteTablet for tablet c07f50fb52904898856af61aab2f4abe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:35:58 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:40624
I20260925 04:35:58.376327 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8491
I20260925 04:35:58.377345 8987 tablet_replica.cc:330] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263: stopping tablet replica
I20260925 04:35:58.377741 8987 raft_consensus.cc:2229] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:35:58.378031 8987 raft_consensus.cc:2258] T c07f50fb52904898856af61aab2f4abe P c0a814c779b046789ec945fd09bbd263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:35:58.378412 8989 tablet_replica.cc:330] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: stopping tablet replica
I20260925 04:35:58.378741 8989 raft_consensus.cc:2229] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:35:58.378990 8989 raft_consensus.cc:2258] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:35:58.386292 8436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:45373 (error 108)
I20260925 04:35:58.387183 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8658
I20260925 04:35:58.389097 8955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:35:58.388104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:45373 (local address 127.0.0.1:55314)
0925 04:35:58.388896 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.388978 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682}
I20260925 04:35:58.389390 8989 ts_tablet_manager.cc:1905] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:35:58.391192 8436 catalog_manager.cc:4584] TS 244242f81a844f9e95042c5fb5c5060d (127.27.46.193:45373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c07f50fb52904898856af61aab2f4abe: Network error: Client connection negotiation failed: client connection to 127.27.46.193:45373: connect: Connection refused (error 111)
I20260925 04:35:58.392527 8989 ts_tablet_manager.cc:1918] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:35:58.392644 8989 log.cc:1196] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/c07f50fb52904898856af61aab2f4abe
I20260925 04:35:58.393013 8989 ts_tablet_manager.cc:1939] T c07f50fb52904898856af61aab2f4abe P f43abcd0252344a7b082a16932c902a8: Deleting consensus metadata
I20260925 04:35:58.400362 8437 catalog_manager.cc:4841] TS f43abcd0252344a7b082a16932c902a8 (127.27.46.195:46025): tablet c07f50fb52904898856af61aab2f4abe (table test-table [id=abbcf8d75cb14398a394ace75292ac42]) successfully deleted
W20260925 04:35:58.404457 8955 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:35:58.404100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:35211 (local address 127.0.0.1:49116)
0925 04:35:58.404232 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 04:35:58.404250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:35:58.404336 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:35211: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 04:35:58.409694 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8793
I20260925 04:35:58.419250 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 8419
I20260925 04:35:58.429226 27835 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:35:58.774335 27835 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:35:58.774466 27835 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:35:59.120671 27835 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:35:59.120 UTC [7475] LOG: received smart shutdown request
2026-09-25 04:35:59.125 UTC [7475] LOG: background worker "logical replication launcher" (PID 7484) exited with exit code 1
2026-09-25 04:35:59.125 UTC [7479] LOG: shutting down
2026-09-25 04:35:59.302 UTC [7475] LOG: database system is shut down
2026-09-25T04:35:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47927 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:35:59 dist-test-slave-r8r3 krb5kdc[8999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:35:59 dist-test-slave-r8r3 krb5kdc[8999](info): set up 2 sockets
Sep 25 04:35:59 dist-test-slave-r8r3 krb5kdc[8999](info): commencing operation
krb5kdc: starting...
W20260925 04:36:00.481926 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:36:00 dist-test-slave-r8r3 krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:36:00.604274 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39955
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:00.716147 9012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:00.716456 9012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:00.716531 9012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:00.720638 9012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:00.720738 9012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:00.720777 9012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:00.720836 9012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:00.720877 9012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:00.726389 9012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:00.727540 9012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:00.728605 9012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:00.737099 9017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:00.737075 9020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:00.737075 9018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:00.737396 9012 server_base.cc:1034] running on GCE node
I20260925 04:36:00.737993 9012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:00.739068 9012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:00.740235 9012 hybrid_clock.cc:648] HybridClock initialized: now 1790310960740214 us; error 29 us; skew 500 ppm
Sep 25 04:36:00 dist-test-slave-r8r3 krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:00.742974 9012 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:00.743935 9012 webserver.cc:460] Webserver started at http://127.27.46.254:46853/ using document root <none> and password file <none>
I20260925 04:36:00.744503 9012 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:00.744597 9012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:00.744858 9012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:00.746973 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "e198eaaaebec4e259a1065d5b7ee07ba"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "b196623684df3c2a112515ef0e2914e0"
server_key_iv: "36580986ae0e007c24fc3c759df4a44f"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.747562 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "e198eaaaebec4e259a1065d5b7ee07ba"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "b196623684df3c2a112515ef0e2914e0"
server_key_iv: "36580986ae0e007c24fc3c759df4a44f"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.751368 9012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 04:36:00.753822 9027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:00.754977 9012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:36:00.755111 9012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "e198eaaaebec4e259a1065d5b7ee07ba"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "b196623684df3c2a112515ef0e2914e0"
server_key_iv: "36580986ae0e007c24fc3c759df4a44f"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.755218 9012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:00.774531 9012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:00.777393 9012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:00.777627 9012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:00.785197 9012 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:39955
I20260925 04:36:00.785234 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:39955 every 8 connection(s)
I20260925 04:36:00.786226 9012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:00.788890 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:00.791198 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9012
I20260925 04:36:00.791316 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:36:00.791666 27835 external_mini_cluster.cc:1428] Setting key 9bbc481caef516003b0f3fc524033eca
I20260925 04:36:00.792975 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.253@KRBTEST.COM" created.
Sep 25 04:36:00 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:00.808429 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.795412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.27.46.254:39955)
0925 04:36:00.795844 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:00.795864 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:00.796149 (+ 285us) server_negotiation.cc:407] Connection header received
0925 04:36:00.797210 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:00.797215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:00.797392 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:00.797606 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:00.798408 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:00.799250 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:00.800361 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:00.800810 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:00.801005 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:00.803534 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:00.803540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:00.803550 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:00.803571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:00.805209 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:00.805762 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:00.805767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:00.805769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:00.805869 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:00.806168 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:00.806173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:00.806175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:00.806537 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:00.806677 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:00.807051 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 04:36:00.807325 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260925 04:36:00.808429 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.795413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39955 (local address 127.0.0.1:56314)
0925 04:36:00.796007 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.796042 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:00.796884 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:00.797598 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:00.797603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:00.798016 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:00.798261 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:00.798274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:00.799419 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:00.799422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:00.800238 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:00.800248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:00.800992 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:00.800996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:00.801160 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:00.801885 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:00.801904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:00.803355 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:00.805340 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:00.805348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:00.805358 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:00.805661 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:00.805974 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:00.805977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:00.805979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:00.806085 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:00.806646 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:00.806652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:00.806810 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 04:36:00.806979 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 04:36:00.807374 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":112,"spinlock_wait_cycles":10496,"thread_start_us":139,"threads_started":1}
I20260925 04:36:00.811269 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:00.811931 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.811681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37506)
0925 04:36:00.811798 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.811866 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 04:36:00.812129 9028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
I20260925 04:36:00.812934 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.812717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37510)
0925 04:36:00.812824 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.812871 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:36:00.813071 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:44147: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
I20260925 04:36:00.817970 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:44147
--webserver_interface=127.27.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:00.862314 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } attempt: 1
I20260925 04:36:00.863229 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.862923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37514)
0925 04:36:00.863079 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.863146 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 04:36:00.863967 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.863780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37516)
0925 04:36:00.863873 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.863915 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 04:36:00.864113 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:44147: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
W20260925 04:36:00.930155 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:00.930433 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:00.930474 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:36:00.932390 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } attempt: 2
I20260925 04:36:00.933367 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.932978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37532)
0925 04:36:00.933170 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.933267 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:36:00.934139 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:00.933927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37536)
0925 04:36:00.934037 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:00.934087 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:36:00.934268 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:44147: Network error: Client connection negotiation failed: client connection to 127.27.46.253:44147: connect: Connection refused (error 111)
W20260925 04:36:00.935017 9088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:00.935092 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:00.935133 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:00.935168 9088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:00.935197 9088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:00.940510 9088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:44147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:00.941758 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:00.942859 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:00.951371 9093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:00.951371 9094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:00.951373 9096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:00.951524 9088 server_base.cc:1034] running on GCE node
I20260925 04:36:00.952170 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:00.953294 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:00.954479 9088 hybrid_clock.cc:648] HybridClock initialized: now 1790310960954466 us; error 65 us; skew 500 ppm
Sep 25 04:36:00 dist-test-slave-r8r3 krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:00.957082 9088 init.cc:377] Logged in from keytab as kudu/127.27.46.253@KRBTEST.COM (short username kudu)
I20260925 04:36:00.958047 9088 webserver.cc:460] Webserver started at http://127.27.46.253:45011/ using document root <none> and password file <none>
I20260925 04:36:00.958688 9088 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:00.958777 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:00.959048 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:00.961143 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data/instance:
uuid: "9b63995af1cb42afb5c75238515e766d"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "9a4be5f95d80577dd02d850a1af33548"
server_key_iv: "a07f0238f1fd07a6a207405d93bc1321"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.961768 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal/instance:
uuid: "9b63995af1cb42afb5c75238515e766d"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "9a4be5f95d80577dd02d850a1af33548"
server_key_iv: "a07f0238f1fd07a6a207405d93bc1321"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.965420 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:00.967783 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:00.968995 9088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:00.969151 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "9b63995af1cb42afb5c75238515e766d"
format_stamp: "Formatted at 2026-09-25 04:36:00 on dist-test-slave-r8r3"
server_key: "9a4be5f95d80577dd02d850a1af33548"
server_key_iv: "a07f0238f1fd07a6a207405d93bc1321"
server_key_version: "encryptionkey@0"
I20260925 04:36:00.969260 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:00.994081 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:00.997288 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:00.997577 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:01.005381 9088 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.253:44147
I20260925 04:36:01.005412 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.253:44147 every 8 connection(s)
I20260925 04:36:01.006433 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
I20260925 04:36:01.009289 9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:01.013190 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.014647 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9088
I20260925 04:36:01.014786 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-1/wal/instance
I20260925 04:36:01.015112 27835 external_mini_cluster.cc:1428] Setting key b061cfd377aa7d57fa07af2030d91f62
WARNING: no policy specified for kudu/127.27.46.252@KRBTEST.COM; defaulting to no policy
I20260925 04:36:01.025413 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.252@KRBTEST.COM" created.
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:36:01.032310 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.015390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.27.46.254:39955)
0925 04:36:01.015561 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.015565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.016634 (+ 1069us) server_negotiation.cc:407] Connection header received
0925 04:36:01.018109 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.018112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.018162 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.018236 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.019503 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.020071 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.021225 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.021458 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.021625 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.026705 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.026709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.026711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.026741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.029356 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.029883 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.029887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.029889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.029968 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.030329 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.030332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.030334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.030534 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.030639 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.031216 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.032093 (+ 877us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:36:01.032546 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.015907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39955 (local address 127.0.0.1:56316)
0925 04:36:01.016467 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.016512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.017804 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.018466 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.018475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.018852 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.019345 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.019358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.020203 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.020206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.021079 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.021088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.021671 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.021675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.021847 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.022450 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.022467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.026543 (+ 4076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.029470 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.029476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.029491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.029781 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.030077 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.030081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.030083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.030227 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.030699 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.030704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.030940 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.031125 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.031478 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":164,"threads_started":1}
I20260925 04:36:01.034368 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.035989 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.025934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37540)
0925 04:36:01.026134 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.026151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.026226 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.027045 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.027049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.027208 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.027381 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.027389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.028352 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.028357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.029018 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.029027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.029934 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.029941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.030102 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.030594 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.030613 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.032083 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.034272 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.034275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.034276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.034489 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.034906 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.034909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.034910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.034995 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.035502 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.035504 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.035574 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.035708 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.035883 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 04:36:01.036129 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.026089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.27.46.253:44147)
0925 04:36:01.026627 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.026639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.026665 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:36:01.026710 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.026718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.026927 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.027062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.027511 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.028223 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.029136 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.029778 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.031297 (+ 1519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.032226 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.032234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.032246 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.032269 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.034160 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.034707 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.034711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.034712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.034781 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.035112 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.035114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.035116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.035417 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.035535 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.035817 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.035959 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":152,"threads_started":1}
I20260925 04:36:01.038044 9081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.038787 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.038537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53432)
0925 04:36:01.038650 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.038709 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:36:01.039529 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.039297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53436)
0925 04:36:01.039431 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.039483 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 04:36:01.039768 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:36:01.043604 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.034889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37554)
0925 04:36:01.035151 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.035164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.035220 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.035443 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.035448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.035636 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.035729 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.035736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.036650 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.036653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.037462 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.037554 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.038606 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.038609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.038691 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.039237 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.039250 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.040555 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.042241 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.042243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.042244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.042432 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.042703 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.042705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.042706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.042777 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.043097 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.043099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.043171 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.043248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.043380 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
I20260925 04:36:01.043672 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.034934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.27.46.253:44147)
0925 04:36:01.035192 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.035198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.035209 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:36:01.035301 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.035304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.035335 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.035401 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.035840 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.036550 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.037672 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.038491 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.038610 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.040690 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.040693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.040695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.040716 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.042138 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.042539 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.042542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.042543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.042625 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.042849 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.042850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.042851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.043027 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.043097 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.043279 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.043432 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
I20260925 04:36:01.044724 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:33001
--webserver_interface=127.27.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:01.044812 9157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.045646 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.045309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53444)
0925 04:36:01.045438 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.045579 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:36:01.045905 9106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
I20260925 04:36:01.046934 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.046647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53456)
0925 04:36:01.046795 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.046860 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:36:01.047096 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
I20260925 04:36:01.070919 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 1
I20260925 04:36:01.071894 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.071543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53468)
0925 04:36:01.071703 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.071766 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 04:36:01.072690 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.072501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53482)
0925 04:36:01.072600 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.072641 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:36:01.072906 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
I20260925 04:36:01.096406 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 1
I20260925 04:36:01.097473 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.097078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53486)
0925 04:36:01.097274 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.097360 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 04:36:01.098296 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.098058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53498)
0925 04:36:01.098169 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.098226 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:36:01.098438 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
I20260925 04:36:01.148069 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 2
I20260925 04:36:01.148931 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.148645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53508)
0925 04:36:01.148792 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.148851 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:36:01.149751 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.149570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53514)
0925 04:36:01.149659 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.149699 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 04:36:01.149999 9081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
W20260925 04:36:01.160605 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:01.160861 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:01.160946 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:01.164857 9164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:01.164916 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:01.164939 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:01.164958 9164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:01.164976 9164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:01.166628 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 2
I20260925 04:36:01.167436 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.167112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53522)
0925 04:36:01.167272 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.167342 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 04:36:01.168143 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.167914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53534)
0925 04:36:01.168015 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.168078 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 04:36:01.168286 9157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:33001: Network error: Client connection negotiation failed: client connection to 127.27.46.252:33001: connect: Connection refused (error 111)
I20260925 04:36:01.170723 9164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:39955,127.27.46.253:44147,127.27.46.252:33001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:01.171787 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:01.172861 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:01.181427 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:01.181519 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:01.181471 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:01.181824 9164 server_base.cc:1034] running on GCE node
I20260925 04:36:01.182367 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:01.183377 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:01.184628 9164 hybrid_clock.cc:648] HybridClock initialized: now 1790310961184595 us; error 40 us; skew 500 ppm
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:01.187580 9164 init.cc:377] Logged in from keytab as kudu/127.27.46.252@KRBTEST.COM (short username kudu)
I20260925 04:36:01.188578 9164 webserver.cc:460] Webserver started at http://127.27.46.252:43991/ using document root <none> and password file <none>
I20260925 04:36:01.189136 9164 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:01.189193 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:01.189427 9164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:01.191601 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data/instance:
uuid: "9458e49a33954f4582189575c9980e85"
format_stamp: "Formatted at 2026-09-25 04:36:01 on dist-test-slave-r8r3"
server_key: "89a0b39b7c839beeba56695e6b0fa85f"
server_key_iv: "b7357d9727d107fe6463b4db44324b74"
server_key_version: "encryptionkey@0"
I20260925 04:36:01.192229 9164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal/instance:
uuid: "9458e49a33954f4582189575c9980e85"
format_stamp: "Formatted at 2026-09-25 04:36:01 on dist-test-slave-r8r3"
server_key: "89a0b39b7c839beeba56695e6b0fa85f"
server_key_iv: "b7357d9727d107fe6463b4db44324b74"
server_key_version: "encryptionkey@0"
I20260925 04:36:01.195948 9164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:36:01.198365 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:01.199635 9164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 04:36:01.199790 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "9458e49a33954f4582189575c9980e85"
format_stamp: "Formatted at 2026-09-25 04:36:01 on dist-test-slave-r8r3"
server_key: "89a0b39b7c839beeba56695e6b0fa85f"
server_key_iv: "b7357d9727d107fe6463b4db44324b74"
server_key_version: "encryptionkey@0"
I20260925 04:36:01.199894 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:01.216578 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:01.219390 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:01.219595 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:01.226979 9164 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.252:33001
I20260925 04:36:01.226989 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.252:33001 every 8 connection(s)
I20260925 04:36:01.227890 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
I20260925 04:36:01.230599 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:01.231930 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9164
I20260925 04:36:01.232064 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790310801918616-27835-0/minicluster-data/master-2/wal/instance
I20260925 04:36:01.232365 27835 external_mini_cluster.cc:1428] Setting key a38a99b156a9b1c4907c437441258275
I20260925 04:36:01.234944 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:01.246130 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.236593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.27.46.254:39955)
0925 04:36:01.236728 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.236732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.236994 (+ 262us) server_negotiation.cc:407] Connection header received
0925 04:36:01.237146 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.237149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.237200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.237283 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.238028 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.238585 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.239493 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.239731 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.239845 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.242768 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.242771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.242772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.242794 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.244206 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.244696 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.244699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.244700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.244785 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.245047 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.245049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.245050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.245266 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.245384 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.245912 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.246011 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:36:01.246920 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.233752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.27.46.252:33001)
0925 04:36:01.234152 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.234168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.234201 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:36:01.234771 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.234786 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.235116 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.235504 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.236582 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.237787 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.238736 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.239254 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.239414 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.242210 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.242219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.242230 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.242255 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.243679 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.244115 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.244126 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.244131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.244230 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.244553 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.244556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.244558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.245071 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.245233 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.245568 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.245928 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":264,"spinlock_wait_cycles":28160,"thread_start_us":144,"threads_started":1}
I20260925 04:36:01.246920 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.236547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:39955 (local address 127.0.0.1:56320)
0925 04:36:01.236878 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.236914 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.237051 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.237332 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.237337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.237757 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.237903 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.237909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.238705 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.238709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.239352 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.239360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.239915 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.239922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.240066 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.240506 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.240530 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.242627 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.244307 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.244318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.244324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.244596 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.244872 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.244875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.244877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.244959 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.245409 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.245416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.245560 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.245705 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.245965 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":116,"spinlock_wait_cycles":2432,"thread_start_us":121,"threads_started":1}
I20260925 04:36:01.248140 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.249193 9081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 3
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:01.257790 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.248587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:44147 (local address 127.0.0.1:37562)
0925 04:36:01.248672 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.248687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.248747 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.249086 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.249091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.249242 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.249357 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.249366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.250483 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.250485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.251326 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.251334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.251910 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.251914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.252014 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.252399 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.252415 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.253851 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.255874 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.255877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.255878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.256193 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.256566 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.256569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.256571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.256685 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.257200 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.257204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.257310 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.257438 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.257596 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":3}
I20260925 04:36:01.257856 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.248733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.27.46.253:44147)
0925 04:36:01.248858 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.248862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.248875 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:36:01.248919 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.248921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.248984 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.249060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.249514 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.250327 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.251456 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.251781 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.251880 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.253991 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.253995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.253997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.254025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.255638 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.256326 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.256332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.256335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.256425 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.256802 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.256806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.256809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.257046 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.257150 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.257531 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.257680 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:36:01.258186 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.249564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53560)
0925 04:36:01.249680 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.249695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.249763 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.250130 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.250133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.250288 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.250358 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.250363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.251414 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.251418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.252067 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.252076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.252700 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.252703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.252781 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.253216 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.253229 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.254486 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.256552 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.256555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.256557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.256853 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.257147 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.257150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.257153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.257240 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.257693 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.257696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.257784 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.257880 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.258035 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:36:01.258692 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.249793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.27.46.252:33001)
0925 04:36:01.249917 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.249921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.249931 (+ 10us) server_negotiation.cc:407] Connection header received
0925 04:36:01.249965 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.249967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.249995 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.250113 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.250475 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.251307 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.252377 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.252582 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.252686 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.254617 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.254621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.254626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.254652 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.256376 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.256968 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.256971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.256973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.257053 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.257343 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.257346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.257348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.257565 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.258370 (+ 805us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.258447 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.258561 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:36:01.259166 9234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:36:01.259393 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.249427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.27.46.254:39955)
0925 04:36:01.249552 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.249559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.249571 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:36:01.249601 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.249603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.249637 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.249700 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.250485 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.251222 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.252069 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.252364 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.252450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.255480 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.255483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.255485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.255507 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.257201 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.257706 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.257709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.257711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.257794 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.258135 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.258138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.258140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.258359 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.258787 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.259146 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.259242 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":3840}
I20260925 04:36:01.259457 9157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } attempt: 3
I20260925 04:36:01.262414 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: Bootstrap starting.
I20260925 04:36:01.265307 9081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:01.266672 9081 log.cc:824] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:36:01 dist-test-slave-r8r3 krb5kdc[8999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:36:01.270532 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.259732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53562)
0925 04:36:01.259883 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.259907 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.260025 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.262628 (+ 2603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.262632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.262781 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.262850 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.262857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.263831 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.263835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.264631 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.264640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.265259 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.265330 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.265778 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.265789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.267167 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.268967 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.268972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.268974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.269201 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.269514 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.269518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.269519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.269611 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.270081 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.270083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.270168 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.270268 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.270385 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 04:36:01.270967 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: No bootstrap required, opened a new log
I20260925 04:36:01.271415 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.261732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.27.46.252:33001)
0925 04:36:01.262373 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.262378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.262390 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:36:01.262418 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.262421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.262457 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.262529 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.262971 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.263713 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.264775 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.265083 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265168 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.267314 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.267318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.267321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.267344 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.268866 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.269302 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.269305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.269307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.269376 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.269709 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.269712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.269713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.269941 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.270050 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.270786 (+ 736us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.271241 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581}
I20260925 04:36:01.272477 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.261981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.27.46.253:44147)
0925 04:36:01.262120 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.262124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.262138 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:36:01.262169 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.262171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.262210 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.262285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.262978 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.263725 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.264784 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.265118 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265179 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.268820 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.268823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.268826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.268849 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.270455 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.270948 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.270952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.270954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.271045 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.271366 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.271370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.271371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.271613 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.271784 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.272181 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.272311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260925 04:36:01.274379 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: Bootstrap starting.
I20260925 04:36:01.274516 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.261851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:33001 (local address 127.0.0.1:53572)
0925 04:36:01.262097 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:01.262123 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:01.262172 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:01.262943 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:01.262945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:01.263096 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:01.263182 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.263188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.264275 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.264278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.265281 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:01.265287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.265932 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:01.266029 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.266437 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:01.266451 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:01.268662 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:01.272854 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.272858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.272860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.273111 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.273449 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:01.273452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:01.273454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:01.273574 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:01.274022 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:01.274025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:01.274120 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 04:36:01.274203 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:36:01.274329 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":276}
I20260925 04:36:01.275092 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.262408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.27.46.252:33001)
0925 04:36:01.262720 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.262727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.262739 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:36:01.262769 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.262771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.262806 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.263281 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.263370 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.264149 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265391 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.265770 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.265942 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.269146 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.269150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.269152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.269179 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.272727 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.273254 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.273258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.273260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.273342 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.273692 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.273697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.273698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.273910 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.274685 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.274762 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.274876 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":11520,"thread_start_us":116,"threads_started":1}
I20260925 04:36:01.276011 9081 raft_consensus.cc:348] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.276276 9081 raft_consensus.cc:374] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:01.276345 9081 raft_consensus.cc:729] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e198eaaaebec4e259a1065d5b7ee07ba, State: Initialized, Role: FOLLOWER
I20260925 04:36:01.276866 9081 consensus_queue.cc:260] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.277122 9234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:01.277861 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: Bootstrap starting.
I20260925 04:36:01.278009 9234 log.cc:824] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:01.278338 9081 sys_catalog.cc:564] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:36:01.279172 9033 tablet.cc:2369] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:01.280332 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: No bootstrap required, opened a new log
I20260925 04:36:01.280422 9157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:01.281260 9157 log.cc:824] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:01.282238 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:36:01.282394 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.282512 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.284494 9234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.284751 9234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:01.284807 9234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9458e49a33954f4582189575c9980e85, State: Initialized, Role: FOLLOWER
I20260925 04:36:01.285315 9234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.285988 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.286216 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.286885 9234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:01.283671 9012 master_runner.cc:186] Error getting master registration for 127.27.46.253:44147: OK, instance_id { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" instance_seqno: 1790310961003125 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:36:01.289830 9012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 04:36:01.291999 9273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:36:01.292079 9273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 04:36:01.292220 9253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:36:01.292299 9253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:36:01.296561 9157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: No bootstrap required, opened a new log
I20260925 04:36:01.299872 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.285638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.27.46.253:44147)
0925 04:36:01.285875 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.285879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.286365 (+ 486us) server_negotiation.cc:407] Connection header received
0925 04:36:01.286488 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.286494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.286549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.286623 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.287738 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.288514 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.289647 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.289918 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.289988 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.294250 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.294253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.294256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.294282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.296560 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.297447 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.297450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.297452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.297555 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.298244 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.298248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.298249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.298468 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.298579 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.299586 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.299723 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260925 04:36:01.296717 9164 master_runner.cc:186] Error getting master registration for 127.27.46.253:44147: OK, instance_id { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" instance_seqno: 1790310961003125 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:36:01.300379 9164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:36:01.300625 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.285660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.27.46.252:33001)
0925 04:36:01.285767 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.285770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.286635 (+ 865us) server_negotiation.cc:407] Connection header received
0925 04:36:01.286680 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.286683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.286725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.286802 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.287747 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.288514 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.291381 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.291709 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.291787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.293911 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.293914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.293917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.293942 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.295828 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.297872 (+ 2044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.297877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.297879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.297959 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.298912 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.298916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.298917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.299127 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.299244 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.300391 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.300528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:36:01.300931 9157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.301167 9157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:01.301245 9157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b63995af1cb42afb5c75238515e766d, State: Initialized, Role: FOLLOWER
I20260925 04:36:01.301805 9157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.303224 9157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:01.310817 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:01.293177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.27.46.254:39955)
0925 04:36:01.293297 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:01.293303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:01.301701 (+ 8398us) server_negotiation.cc:407] Connection header received
0925 04:36:01.301966 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:01.301971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:01.302026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:01.302076 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:01.303052 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.303738 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.304818 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:01.305203 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:01.305317 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:01.306847 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:01.306851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:01.306853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:01.306876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:01.308576 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.309045 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.309049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.309051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.309141 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:01.309464 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:01.309467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:01.309469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:01.309723 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:01.309854 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:01.310508 (+ 654us) server_negotiation.cc:299] Negotiation successful
0925 04:36:01.310633 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:36:01.310966 9277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.311090 9277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.311605 9277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:01.311757 9277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.312000 9088 master_runner.cc:263] No leader master found from master 9b63995af1cb42afb5c75238515e766d
I20260925 04:36:01.312062 9088 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 04:36:01.312630 9288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:36:01.312707 9288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:36:01.313989 9208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b63995af1cb42afb5c75238515e766d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9458e49a33954f4582189575c9980e85" is_pre_election: true
I20260925 04:36:01.314028 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b63995af1cb42afb5c75238515e766d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" is_pre_election: true
I20260925 04:36:01.314262 9208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b63995af1cb42afb5c75238515e766d in term 0.
I20260925 04:36:01.314313 9054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b63995af1cb42afb5c75238515e766d in term 0.
I20260925 04:36:01.314625 9106 leader_election.cc:302] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9458e49a33954f4582189575c9980e85, 9b63995af1cb42afb5c75238515e766d; no voters:
I20260925 04:36:01.314884 9277 leader_election.cc:288] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e198eaaaebec4e259a1065d5b7ee07ba (127.27.46.254:39955), 9458e49a33954f4582189575c9980e85 (127.27.46.252:33001)
I20260925 04:36:01.315019 9277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:36:01.315116 9277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:01.315177 9277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:01.316548 9277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.317196 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b63995af1cb42afb5c75238515e766d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e198eaaaebec4e259a1065d5b7ee07ba"
I20260925 04:36:01.317328 9054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:01.318336 9054 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b63995af1cb42afb5c75238515e766d in term 1.
I20260925 04:36:01.318691 9105 leader_election.cc:302] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b63995af1cb42afb5c75238515e766d, e198eaaaebec4e259a1065d5b7ee07ba; no voters:
I20260925 04:36:01.325615 9290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:01.325850 9290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [term 1 LEADER]: Becoming Leader. State: Replica: 9b63995af1cb42afb5c75238515e766d, State: Running, Role: LEADER
I20260925 04:36:01.326056 9208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b63995af1cb42afb5c75238515e766d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9458e49a33954f4582189575c9980e85"
I20260925 04:36:01.326171 9208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:01.326165 9290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } }
I20260925 04:36:01.327353 9208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b63995af1cb42afb5c75238515e766d in term 1.
I20260925 04:36:01.327767 9290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b63995af1cb42afb5c75238515e766d. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.327860 9290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:01.329757 9277 leader_election.cc:288] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [CANDIDATE]: Term 1 election: Requested vote from peers e198eaaaebec4e259a1065d5b7ee07ba (127.27.46.254:39955), 9458e49a33954f4582189575c9980e85 (127.27.46.252:33001)
I20260925 04:36:01.331542 9293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:01.332573 9293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:01.340509 9208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [term 1 FOLLOWER]: Refusing update from remote peer 9b63995af1cb42afb5c75238515e766d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:01.341281 9291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:01.345257 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b63995af1cb42afb5c75238515e766d. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.345366 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.346457 9054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [term 1 FOLLOWER]: Refusing update from remote peer 9b63995af1cb42afb5c75238515e766d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:01.346992 9291 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:01.351780 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b63995af1cb42afb5c75238515e766d. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.351900 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.353530 9290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.353636 9290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:01.353740 9294 mvcc.cc:205] Tried to move back new op lower bound from 7333113697640955904 to 7333113697594732544. Current Snapshot: MvccSnapshot[applied={T|T < 7333113697640955904}]
I20260925 04:36:01.353871 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.353988 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.356703 9277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.356799 9277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:01.358383 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b63995af1cb42afb5c75238515e766d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e198eaaaebec4e259a1065d5b7ee07ba" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 39955 } } peers { permanent_uuid: "9b63995af1cb42afb5c75238515e766d" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 44147 } } peers { permanent_uuid: "9458e49a33954f4582189575c9980e85" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 33001 } } }
I20260925 04:36:01.358503 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:36:01.363912 9293 catalog_manager.cc:1347] Generated new cluster ID: 7d7d83e29022426dba03975c566be89e
I20260925 04:36:01.363971 9293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:01.394868 9293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:36:01.396178 9293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:01.412209 9293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b63995af1cb42afb5c75238515e766d: Generated new TSK 0
I20260925 04:36:01.413793 9293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:36:02.293385 9273 catalog_manager.cc:1259] Loaded cluster ID: 7d7d83e29022426dba03975c566be89e
I20260925 04:36:02.293474 9273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: loading cluster ID for follower catalog manager: success
I20260925 04:36:02.294063 9253 catalog_manager.cc:1259] Loaded cluster ID: 7d7d83e29022426dba03975c566be89e
I20260925 04:36:02.294147 9253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: loading cluster ID for follower catalog manager: success
I20260925 04:36:02.295305 9273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: acquiring CA information for follower catalog manager: success
I20260925 04:36:02.295742 9253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: acquiring CA information for follower catalog manager: success
I20260925 04:36:02.296531 9273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9458e49a33954f4582189575c9980e85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:36:02.296810 9253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e198eaaaebec4e259a1065d5b7ee07ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:36:02.484367 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9012
I20260925 04:36:02.491966 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9088
I20260925 04:36:02.498643 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9164
2026-09-25T04:36:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3069 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:36:02 dist-test-slave-r8r3 krb5kdc[9309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:36:02 dist-test-slave-r8r3 krb5kdc[9309](info): set up 2 sockets
Sep 25 04:36:02 dist-test-slave-r8r3 krb5kdc[9309](info): commencing operation
krb5kdc: starting...
W20260925 04:36:03.521353 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:36:03.643718 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34019
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:34019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:03.753409 9322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:03.753727 9322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:03.753803 9322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:03.757606 9322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:03.757702 9322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:03.757740 9322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:03.757799 9322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:03.757853 9322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:03.762939 9322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:34019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:03.764078 9322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:03.765127 9322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:03.773420 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:03.773428 9327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:03.773463 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:03.773609 9322 server_base.cc:1034] running on GCE node
I20260925 04:36:03.774271 9322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:03.775384 9322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:03.776553 9322 hybrid_clock.cc:648] HybridClock initialized: now 1790310963776525 us; error 40 us; skew 500 ppm
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310963, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:03.779222 9322 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:03.780103 9322 webserver.cc:460] Webserver started at http://127.27.46.254:34149/ using document root <none> and password file <none>
I20260925 04:36:03.780690 9322 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:03.780751 9322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:03.781013 9322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:03.783142 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1"
format_stamp: "Formatted at 2026-09-25 04:36:03 on dist-test-slave-r8r3"
server_key: "8cf85fbe12204eaf9fabe237655d7916"
server_key_iv: "e09c85fef62a9ceabb2af5e2bf489f3f"
server_key_version: "encryptionkey@0"
I20260925 04:36:03.783746 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1"
format_stamp: "Formatted at 2026-09-25 04:36:03 on dist-test-slave-r8r3"
server_key: "8cf85fbe12204eaf9fabe237655d7916"
server_key_iv: "e09c85fef62a9ceabb2af5e2bf489f3f"
server_key_version: "encryptionkey@0"
I20260925 04:36:03.787509 9322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:36:03.789831 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:03.791085 9322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:03.791213 9322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1"
format_stamp: "Formatted at 2026-09-25 04:36:03 on dist-test-slave-r8r3"
server_key: "8cf85fbe12204eaf9fabe237655d7916"
server_key_iv: "e09c85fef62a9ceabb2af5e2bf489f3f"
server_key_version: "encryptionkey@0"
I20260925 04:36:03.791318 9322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:03.808118 9322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:03.811194 9322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:03.811405 9322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:03.818444 9322 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:34019
I20260925 04:36:03.818470 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:34019 every 8 connection(s)
I20260925 04:36:03.819353 9322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:03.820199 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9322
I20260925 04:36:03.820310 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:36:03.820639 27835 external_mini_cluster.cc:1428] Setting key a6d27594380a6485b581c81d4f77533c
I20260925 04:36:03.822579 9391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:03.829257 9391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1: Bootstrap starting.
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:03.832015 9391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:03.832937 9391 log.cc:824] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:03.835194 9391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1: No bootstrap required, opened a new log
I20260925 04:36:03.835547 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:03.822037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.27.46.254:34019)
0925 04:36:03.822448 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:03.822466 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:03.822505 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:36:03.823349 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:03.823372 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:03.823825 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:03.824190 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:03.824873 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:03.826406 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:03.827264 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:03.827913 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:03.828068 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:03.830600 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:03.830606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:03.830623 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:03.830656 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:03.832620 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:03.833058 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:03.833063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:03.833069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:03.833205 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:03.833484 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:03.833488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:03.833489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:03.833987 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:03.834143 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:03.834436 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 04:36:03.834765 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260925 04:36:03.838587 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:36:03.839462 9391 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } }
I20260925 04:36:03.839680 9391 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:03.839763 9391 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb3f3ecbab643fe846ddfb9b9f81cd1, State: Initialized, Role: FOLLOWER
I20260925 04:36:03.840215 9391 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } }
I20260925 04:36:03.840415 9391 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:03.840468 9391 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:03.840541 9391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:03.841372 9391 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } }
I20260925 04:36:03.841774 9391 leader_election.cc:302] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb3f3ecbab643fe846ddfb9b9f81cd1; no voters:
I20260925 04:36:03.842077 9391 leader_election.cc:288] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:36:03.842247 9396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:03.842563 9396 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [term 1 LEADER]: Becoming Leader. State: Replica: dcb3f3ecbab643fe846ddfb9b9f81cd1, State: Running, Role: LEADER
I20260925 04:36:03.842877 9396 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } }
I20260925 04:36:03.843289 9391 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:03.844524 9397 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } } }
I20260925 04:36:03.844658 9397 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:03.845028 9398 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb3f3ecbab643fe846ddfb9b9f81cd1. Latest consensus state: current_term: 1 leader_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb3f3ecbab643fe846ddfb9b9f81cd1" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34019 } } }
I20260925 04:36:03.845166 9405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:03.845224 9398 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:03.848446 9405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:03.853973 9405 catalog_manager.cc:1347] Generated new cluster ID: a27770fe399544e5bc5d38fcc6197fcd
I20260925 04:36:03.854070 9405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:03.866420 9405 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:36:03.867751 9405 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:03.879949 9405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb3f3ecbab643fe846ddfb9b9f81cd1: Generated new TSK 0
I20260925 04:36:03.880828 9405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:36:03.894701 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9322
2026-09-25T04:36:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1394 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9419](info): set up 2 sockets
Sep 25 04:36:03 dist-test-slave-r8r3 krb5kdc[9419](info): commencing operation
krb5kdc: starting...
W20260925 04:36:04.909091 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:36:04 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:36:05.031994 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:05.142594 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:05.142876 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:05.142920 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:05.146925 9432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:05.146976 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:05.146994 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:05.147011 9432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:05.147038 9432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:05.153168 9432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:05.154241 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:05.155237 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:05.164234 9440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.164249 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.164444 9432 server_base.cc:1034] running on GCE node
W20260925 04:36:05.164250 9438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.165046 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:05.166222 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:05.167411 9432 hybrid_clock.cc:648] HybridClock initialized: now 1790310965167382 us; error 40 us; skew 500 ppm
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:05.174274 9432 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:05.175185 9432 webserver.cc:460] Webserver started at http://127.27.46.254:37233/ using document root <none> and password file <none>
I20260925 04:36:05.175762 9432 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:05.175868 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:05.176100 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:05.178361 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "6c668e49f7db4bc7beefc7ddbef5004d"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "25367748b8c449558f7dd2d668be4e32"
server_key_iv: "a18e4c0cd9c64b3977415439a37d79fc"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.178920 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "6c668e49f7db4bc7beefc7ddbef5004d"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "25367748b8c449558f7dd2d668be4e32"
server_key_iv: "a18e4c0cd9c64b3977415439a37d79fc"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.182588 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:36:05.184940 9447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.186172 9432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:05.186326 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "6c668e49f7db4bc7beefc7ddbef5004d"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "25367748b8c449558f7dd2d668be4e32"
server_key_iv: "a18e4c0cd9c64b3977415439a37d79fc"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.186437 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:05.209403 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:05.212675 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:05.212909 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:05.220521 9432 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:46313
I20260925 04:36:05.220814 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:46313 every 8 connection(s)
I20260925 04:36:05.221710 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:05.224632 9500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:05.228626 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9432
I20260925 04:36:05.228765 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:36:05.229104 27835 external_mini_cluster.cc:1428] Setting key 0f1c5d6292ee637fa557f8fc42946418
I20260925 04:36:05.230948 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap starting.
I20260925 04:36:05.233238 9500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:05.234222 9500 log.cc:824] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:05.236478 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: No bootstrap required, opened a new log
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:05.240615 9500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:05.240844 9500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:05.240921 9500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Initialized, Role: FOLLOWER
I20260925 04:36:05.241432 9500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:05.241631 9500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:05.241701 9500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:05.241832 9500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:05.242762 9500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:05.243122 9500 leader_election.cc:302] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c668e49f7db4bc7beefc7ddbef5004d; no voters:
I20260925 04:36:05.243485 9500 leader_election.cc:288] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:36:05.243592 9505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:05.243870 9505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 LEADER]: Becoming Leader. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Running, Role: LEADER
I20260925 04:36:05.244266 9505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:05.244879 9500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:05.244992 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.230664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60034 (local address 127.27.46.254:46313)
0925 04:36:05.231203 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.231221 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.231258 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:36:05.231907 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.231924 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.232266 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.232627 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:05.233479 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.234926 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.235938 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.236612 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.236774 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.239664 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:05.239670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:05.239683 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:05.239711 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:05.241652 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.242197 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.242203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.242209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.242342 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.242635 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.242639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.242641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.243118 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:05.243326 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.243684 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.244017 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":163,"threads_started":1}
I20260925 04:36:05.246028 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:05.246153 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:05.246397 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c668e49f7db4bc7beefc7ddbef5004d. Latest consensus state: current_term: 1 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:05.246472 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:05.247635 9514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:05.250499 9514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:05.256101 9514 catalog_manager.cc:1347] Generated new cluster ID: 2e8a03a7a848432a9582759258b50ef0
I20260925 04:36:05.256165 9514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:05.263243 9514 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:36:05.264456 9514 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:05.270859 9514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Generated new TSK 0
I20260925 04:36:05.271611 9514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:36:05.310441 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:05.425904 9526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:05.426222 9526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:05.426316 9526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:05.430303 9526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:05.430435 9526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:05.430529 9526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:36:05.435755 9526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:05.436915 9526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:05.438028 9526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:05.448117 9531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.448160 9532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.448156 9534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.448378 9526 server_base.cc:1034] running on GCE node
I20260925 04:36:05.448829 9526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:05.449407 9526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:05.450574 9526 hybrid_clock.cc:648] HybridClock initialized: now 1790310965450558 us; error 25 us; skew 500 ppm
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:05.453243 9526 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:36:05.454207 9526 webserver.cc:460] Webserver started at http://127.27.46.193:44309/ using document root <none> and password file <none>
I20260925 04:36:05.454793 9526 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:05.454862 9526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:05.455101 9526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:05.457173 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "90a7b4a3ba3ca1a16653c09c0e54b24d"
server_key_iv: "409ad6de8818a9fd6e496623ebd5dfb1"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.457808 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "90a7b4a3ba3ca1a16653c09c0e54b24d"
server_key_iv: "409ad6de8818a9fd6e496623ebd5dfb1"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.461609 9526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:36:05.464125 9541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.465464 9526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:05.465643 9526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "90a7b4a3ba3ca1a16653c09c0e54b24d"
server_key_iv: "409ad6de8818a9fd6e496623ebd5dfb1"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.465752 9526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:05.486202 9526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:05.489341 9526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:05.489657 9526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:05.490350 9526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:05.491233 9526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:05.491286 9526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.491375 9526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:05.491410 9526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.500422 9526 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:41927
I20260925 04:36:05.500497 9657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:41927 every 8 connection(s)
I20260925 04:36:05.501384 9526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:36:05.508432 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9526
I20260925 04:36:05.508571 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:36:05.508913 27835 external_mini_cluster.cc:1428] Setting key ba8d9e8990168b8b4c79eab6247e9867
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:05.516016 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.503309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:34101 (local address 127.27.46.254:46313)
0925 04:36:05.503504 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.503510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.504390 (+ 880us) server_negotiation.cc:407] Connection header received
0925 04:36:05.505757 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.505759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.505812 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.505885 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:05.507126 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.507687 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.509181 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.509417 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.509561 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.512192 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:05.512196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:05.512198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:05.512232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:05.513812 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.514352 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.514357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.514359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.514444 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.514866 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.514870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.514872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.515057 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:05.515162 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.515679 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.515840 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:36:05.516844 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.503753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:34101)
0925 04:36:05.504218 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.504265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.505561 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.506061 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.506068 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.506480 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:05.506975 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.506990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.507797 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.507799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.509062 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.509075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.509601 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.509606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.509820 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.510529 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:05.510553 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:05.512030 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:05.513931 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.513937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.513953 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.514237 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.514538 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.514542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.514544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.514765 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.515177 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:05.515185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:05.515444 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.515639 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.515981 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260925 04:36:05.518296 9658 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:36:05.518573 9658 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:05.519084 9658 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:05.520671 9464 ts_manager.cc:194] Registered new tserver with Master: 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:36:05.521981 9464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:34101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:36:05.536844 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:05.651649 9663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:05.651921 9663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:05.651965 9663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:05.656036 9663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:05.656100 9663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:05.656159 9663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:36:05.661361 9663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:05.662490 9663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:05.663532 9663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:05.673291 9668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.673372 9669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.673424 9663 server_base.cc:1034] running on GCE node
W20260925 04:36:05.673291 9671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.673995 9663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:05.674654 9663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:05.675835 9663 hybrid_clock.cc:648] HybridClock initialized: now 1790310965675810 us; error 40 us; skew 500 ppm
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:05.678530 9663 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:36:05.679450 9663 webserver.cc:460] Webserver started at http://127.27.46.194:39789/ using document root <none> and password file <none>
I20260925 04:36:05.680022 9663 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:05.680114 9663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:05.680359 9663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:05.682482 9663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "c25cf5a5e65840d393e66b84603d3fbe"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "ed664f899c5afd25b8243207deffbf26"
server_key_iv: "e29a49d2f6d804cec00155ff9da46953"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.683053 9663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "c25cf5a5e65840d393e66b84603d3fbe"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "ed664f899c5afd25b8243207deffbf26"
server_key_iv: "e29a49d2f6d804cec00155ff9da46953"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.686729 9663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:36:05.689222 9678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.690567 9663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:05.690701 9663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c25cf5a5e65840d393e66b84603d3fbe"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "ed664f899c5afd25b8243207deffbf26"
server_key_iv: "e29a49d2f6d804cec00155ff9da46953"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.690814 9663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:05.706195 9663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:05.709185 9663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:05.709411 9663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:05.710049 9663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:05.710875 9663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:05.710944 9663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.711015 9663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:05.711066 9663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.720404 9663 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37097
I20260925 04:36:05.720432 9792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37097 every 8 connection(s)
I20260925 04:36:05.721305 9663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:36:05.723690 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9663
I20260925 04:36:05.723824 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:36:05.724159 27835 external_mini_cluster.cc:1428] Setting key c74c65a3b670d70f920e182df4d5950c
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:36:05.735918 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.723246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:55663 (local address 127.27.46.254:46313)
0925 04:36:05.723390 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.723395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.724447 (+ 1052us) server_negotiation.cc:407] Connection header received
0925 04:36:05.726230 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.726233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.726285 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.726367 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:05.727775 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.728302 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.729469 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.729741 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.729850 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.732287 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:05.732290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:05.732293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:05.732326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:05.733836 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.734298 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.734304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.734308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.734427 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.734739 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.734742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.734744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.734917 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:05.735025 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.735624 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.735747 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:36:05.736869 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.723744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:55663)
0925 04:36:05.724292 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.724345 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.726028 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.726562 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.726569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.726922 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:05.727597 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.727619 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.728419 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.728422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.729317 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.729326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.729887 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.729899 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.730068 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.730711 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:05.730728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:05.732128 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:05.733943 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.733949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.733961 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.734199 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.734507 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.734511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.734512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.734645 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.735025 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:05.735033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:05.735301 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.735579 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.735902 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260925 04:36:05.738123 9793 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
I20260925 04:36:05.738384 9793 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:05.739089 9793 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:05.740159 9464 ts_manager.cc:194] Registered new tserver with Master: c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097)
I20260925 04:36:05.740737 9464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:55663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:36:05.752592 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:05.865872 9798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:05.866151 9798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:05.866194 9798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:05.870471 9798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:05.870529 9798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:05.870635 9798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:36:05.875931 9798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:05.877061 9798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:05.878228 9798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:05.888010 9804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.888049 9805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:05.888049 9807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:05.888247 9798 server_base.cc:1034] running on GCE node
I20260925 04:36:05.888830 9798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:05.889582 9798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:05.890789 9798 hybrid_clock.cc:648] HybridClock initialized: now 1790310965890765 us; error 39 us; skew 500 ppm
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:05.893582 9798 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:36:05.894423 9798 webserver.cc:460] Webserver started at http://127.27.46.195:34473/ using document root <none> and password file <none>
I20260925 04:36:05.895057 9798 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:05.895119 9798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:05.895303 9798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:05.897320 9798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "fb45a4c0b9324c8f821b4e85142fc8ae"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "263709563e2e7d124dd14929a4c24404"
server_key_iv: "41c8157227478b085fb6b219f0c7ceee"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.897899 9798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "fb45a4c0b9324c8f821b4e85142fc8ae"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "263709563e2e7d124dd14929a4c24404"
server_key_iv: "41c8157227478b085fb6b219f0c7ceee"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.901461 9798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:05.903899 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.905144 9798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:05.905241 9798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "fb45a4c0b9324c8f821b4e85142fc8ae"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "263709563e2e7d124dd14929a4c24404"
server_key_iv: "41c8157227478b085fb6b219f0c7ceee"
server_key_version: "encryptionkey@0"
I20260925 04:36:05.905324 9798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:05.923380 9798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:05.926291 9798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:05.926453 9798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:05.927003 9798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:05.927801 9798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:05.927848 9798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.927896 9798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:05.927917 9798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:05.936439 9798 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:40643
I20260925 04:36:05.936460 9928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:40643 every 8 connection(s)
I20260925 04:36:05.937542 9798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:36:05.939544 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9798
I20260925 04:36:05.939689 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:36:05.939987 27835 external_mini_cluster.cc:1428] Setting key 0c1d237c1404573867fb63038ee86e2e
Sep 25 04:36:05 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310965, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:05.951524 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.939513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:60803 (local address 127.27.46.254:46313)
0925 04:36:05.939699 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.939706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.940688 (+ 982us) server_negotiation.cc:407] Connection header received
0925 04:36:05.942500 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.942502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.942546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.942611 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:05.943734 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.944244 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.945406 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.945704 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.945784 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.948340 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:05.948343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:05.948345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:05.948371 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:05.949633 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.950186 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.950189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.950190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.950267 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.950568 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.950570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.950571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.950742 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:05.950816 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.951290 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.951399 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:36:05.952459 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.939929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:60803)
0925 04:36:05.940503 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.940548 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.942259 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.942749 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.942755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.943114 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:05.943612 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.943626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.944351 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.944353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.945291 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.945300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.945863 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.945866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.946016 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.946690 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:05.946707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:05.948182 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:05.949733 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.949737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.949771 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.950097 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.950349 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:05.950351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:05.950356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:05.950489 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:05.950839 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:05.950844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:05.951079 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.951243 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.951501 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":145,"threads_started":1}
I20260925 04:36:05.953764 9929 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
I20260925 04:36:05.954007 9929 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:05.954509 9929 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:05.955545 9463 ts_manager.cc:194] Registered new tserver with Master: fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643)
I20260925 04:36:05.956111 9463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:60803
I20260925 04:36:05.965915 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:36:05.975087 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.967905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.27.46.254:46313)
0925 04:36:05.968027 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.968032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.968206 (+ 174us) server_negotiation.cc:407] Connection header received
0925 04:36:05.968313 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.968316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.968365 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.968437 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:05.969116 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.969775 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.970640 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.970846 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.970927 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.972176 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:05.972179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:05.972181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:05.972203 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:05.973685 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.974041 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.974043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.974044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.974113 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:05.974361 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:05.974363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:05.974364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:05.974577 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:05.974681 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.974872 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.974968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:36:05.978096 9463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:36:05.981122 9463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:36:05.996031 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.990524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.0.0.1:34310)
0925 04:36:05.991178 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.991196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.991266 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.991752 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.991756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.991774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:05.991873 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.991879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.993003 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.993007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.994473 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.994484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.995448 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.995452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.995556 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.995592 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.995729 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.995861 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260925 04:36:05.996037 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.990945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.27.46.193:41927)
0925 04:36:05.991338 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.991354 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.991375 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:36:05.991448 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.991456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.991635 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.991779 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:05.991991 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.992870 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.994608 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.995320 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.995449 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.995611 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.995760 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.995869 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
I20260925 04:36:05.996260 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.989963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.0.0.1:44612)
0925 04:36:05.990765 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.990780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.990929 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.992056 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.992059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.992078 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:05.992152 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.992160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.993433 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.993436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.994691 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.994698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.995846 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.995849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.995948 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.995970 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.996022 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.996093 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":154,"spinlock_wait_cycles":5376,"thread_start_us":103,"threads_started":1}
I20260925 04:36:05.996618 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.989920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.27.46.195:40643)
0925 04:36:05.990551 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.990565 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.991078 (+ 513us) server_negotiation.cc:407] Connection header received
0925 04:36:05.991161 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.991166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.991383 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.991553 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:05.991886 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.993104 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.994944 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.995700 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.995830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.995976 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.996349 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.996455 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":110,"threads_started":1}
I20260925 04:36:05.996447 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.989770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:40643 (local address 127.0.0.1:41210)
0925 04:36:05.990309 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:05.990769 (+ 460us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:05.990929 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:05.991571 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:05.991577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:05.991603 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:05.991744 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.991751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.993280 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.993283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.994668 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:05.994679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.996036 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.996040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:05.996123 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.996142 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 04:36:05.996205 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 04:36:05.996262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":105,"spinlock_wait_cycles":4736,"thread_start_us":112,"threads_started":1}
I20260925 04:36:05.996850 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:05.990469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.27.46.194:37097)
0925 04:36:05.991609 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:05.991622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:05.991645 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:36:05.991716 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:05.991721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:05.991927 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:05.992068 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:05.992252 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.993297 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.994875 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:05.995700 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:05.996392 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:05.996528 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:05.996579 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:36:05.996686 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":112,"threads_started":1}
I20260925 04:36:05.999640 9727 tablet_service.cc:1467] Processing CreateTablet for tablet 411600d4cde9442a8f74a58dedf3a670 (DEFAULT_TABLE table=test-table [id=20db3788e58b46de85877eddccc33795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:05.999938 9590 tablet_service.cc:1467] Processing CreateTablet for tablet 411600d4cde9442a8f74a58dedf3a670 (DEFAULT_TABLE table=test-table [id=20db3788e58b46de85877eddccc33795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:05.999936 9862 tablet_service.cc:1467] Processing CreateTablet for tablet 411600d4cde9442a8f74a58dedf3a670 (DEFAULT_TABLE table=test-table [id=20db3788e58b46de85877eddccc33795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:06.000756 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411600d4cde9442a8f74a58dedf3a670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:06.000998 9590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411600d4cde9442a8f74a58dedf3a670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:06.000999 9862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411600d4cde9442a8f74a58dedf3a670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:06.007359 9947 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Bootstrap starting.
I20260925 04:36:06.008921 9948 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Bootstrap starting.
I20260925 04:36:06.009898 9949 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap starting.
I20260925 04:36:06.011096 9947 tablet_bootstrap.cc:654] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:06.011178 9948 tablet_bootstrap.cc:654] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:06.011636 9949 tablet_bootstrap.cc:654] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:06.012100 9947 log.cc:824] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:06.012414 9948 log.cc:824] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:06.012791 9949 log.cc:824] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:06.014415 9947 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: No bootstrap required, opened a new log
I20260925 04:36:06.014416 9948 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: No bootstrap required, opened a new log
I20260925 04:36:06.014622 9948 ts_tablet_manager.cc:1397] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:36:06.014622 9947 ts_tablet_manager.cc:1397] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:36:06.018699 9947 raft_consensus.cc:348] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.018699 9948 raft_consensus.cc:348] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.019001 9947 raft_consensus.cc:374] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:06.019001 9948 raft_consensus.cc:374] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:06.019057 9947 raft_consensus.cc:729] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25cf5a5e65840d393e66b84603d3fbe, State: Initialized, Role: FOLLOWER
I20260925 04:36:06.019068 9948 raft_consensus.cc:729] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb45a4c0b9324c8f821b4e85142fc8ae, State: Initialized, Role: FOLLOWER
I20260925 04:36:06.019591 9948 consensus_queue.cc:260] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.019591 9947 consensus_queue.cc:260] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.020295 9929 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
I20260925 04:36:06.020521 9793 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
I20260925 04:36:06.020668 9948 ts_tablet_manager.cc:1428] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:36:06.021770 9947 ts_tablet_manager.cc:1428] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 04:36:06.023093 9949 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: No bootstrap required, opened a new log
I20260925 04:36:06.023393 9949 ts_tablet_manager.cc:1397] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260925 04:36:06.027066 9949 raft_consensus.cc:348] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.027297 9949 raft_consensus.cc:374] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:06.027374 9949 raft_consensus.cc:729] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, State: Initialized, Role: FOLLOWER
I20260925 04:36:06.027796 9949 consensus_queue.cc:260] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.028438 9658 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
I20260925 04:36:06.028784 9949 ts_tablet_manager.cc:1428] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 04:36:06.188594 9930 tablet.cc:2369] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:06.199239 9954 raft_consensus.cc:482] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:06.199486 9954 raft_consensus.cc:504] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.200790 9954 leader_election.cc:288] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097), 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:06.206818 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.201091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.27.46.195:39129)
0925 04:36:06.202270 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.202286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.202450 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.202917 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.202919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.202936 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:06.203083 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.203091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.204162 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.204165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.205653 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.205663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.206363 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.206367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.206464 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.206487 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.206567 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.206643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":170,"thread_start_us":656,"threads_started":1}
I20260925 04:36:06.206802 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.201249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:39129 (local address 127.27.46.193:41927)
0925 04:36:06.201381 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.201386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.202369 (+ 983us) server_negotiation.cc:407] Connection header received
0925 04:36:06.202582 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.202585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.202658 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.202766 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:06.203250 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.204019 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.205803 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.206199 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.206358 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.206418 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.206597 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.206689 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:36:06.207351 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.201091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.27.46.195:49915)
0925 04:36:06.201243 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.201262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.201372 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.201834 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.201839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.201875 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:06.202043 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.202054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.203434 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.203437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.205240 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.205251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.206581 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.206588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.206759 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.206795 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.206873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.206963 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":57}
I20260925 04:36:06.207688 9610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" is_pre_election: true
I20260925 04:36:06.207893 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.201381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:49915 (local address 127.27.46.194:37097)
0925 04:36:06.201542 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.201546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.201562 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:36:06.201622 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.201624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.201679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.201786 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:06.202210 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.203264 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.205363 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.206322 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.207546 (+ 1224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.207618 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.207662 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.207756 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:36:06.208097 9610 raft_consensus.cc:2454] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 0.
I20260925 04:36:06.208709 9815 leader_election.cc:302] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, fb45a4c0b9324c8f821b4e85142fc8ae; no voters:
I20260925 04:36:06.208942 9747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe" is_pre_election: true
I20260925 04:36:06.209434 9747 raft_consensus.cc:2454] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 0.
I20260925 04:36:06.212052 9954 raft_consensus.cc:2781] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:36:06.212152 9954 raft_consensus.cc:482] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:06.212188 9954 raft_consensus.cc:3037] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:06.213658 9955 raft_consensus.cc:482] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:06.213671 9954 raft_consensus.cc:504] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.213785 9955 raft_consensus.cc:504] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.214746 9954 leader_election.cc:288] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 election: Requested vote from peers c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097), 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:06.215029 9610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
I20260925 04:36:06.215133 9955 leader_election.cc:288] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643), c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097)
I20260925 04:36:06.215166 9610 raft_consensus.cc:3037] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:06.215992 9747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe"
I20260925 04:36:06.216141 9747 raft_consensus.cc:3037] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:06.216853 9610 raft_consensus.cc:2454] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 1.
I20260925 04:36:06.218086 9747 raft_consensus.cc:2454] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 1.
I20260925 04:36:06.218449 9817 leader_election.cc:302] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25cf5a5e65840d393e66b84603d3fbe, fb45a4c0b9324c8f821b4e85142fc8ae; no voters:
I20260925 04:36:06.218703 9954 raft_consensus.cc:2781] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:06.218953 9954 raft_consensus.cc:686] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 LEADER]: Becoming Leader. State: Replica: fb45a4c0b9324c8f821b4e85142fc8ae, State: Running, Role: LEADER
I20260925 04:36:06.219353 9954 consensus_queue.cc:237] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.221930 9944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.216373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:41843 (local address 127.27.46.194:37097)
0925 04:36:06.216498 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.216504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.216929 (+ 425us) server_negotiation.cc:407] Connection header received
0925 04:36:06.217051 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.217054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.217096 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.217187 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:06.217488 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.218035 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.219720 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.220380 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.220557 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.220643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.221733 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.221815 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:36:06.222277 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.216524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.27.46.193:41843)
0925 04:36:06.216822 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.216836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.216968 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.217199 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.217202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.217217 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:06.217350 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.217360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.218154 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.218157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.219419 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.219430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.221331 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.221335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.221471 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.221592 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.222044 (+ 452us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.222120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":79,"thread_start_us":111,"threads_started":1}
W20260925 04:36:06.222556 9794 tablet.cc:2369] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:06.224002 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.219803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:56209 (local address 127.27.46.195:40643)
0925 04:36:06.219946 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.219951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.219966 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:36:06.220008 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.220012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.220054 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.220122 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:06.220515 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.221160 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.222802 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.223392 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.223521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.223566 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.223818 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.223893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:36:06.224114 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.215352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:40643 (local address 127.27.46.193:56209)
0925 04:36:06.215641 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.215656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.215763 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.220233 (+ 4470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.220236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.220252 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:06.220368 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.220374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.221304 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.221309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.222599 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.222612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.223549 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.223554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.223668 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.223714 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.223805 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.223912 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260925 04:36:06.224764 9882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" is_pre_election: true
I20260925 04:36:06.226202 9747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "411600d4cde9442a8f74a58dedf3a670" candidate_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe" is_pre_election: true
I20260925 04:36:06.226418 9747 raft_consensus.cc:2379] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e1f54acfd3f4ac9bf9e9fefd10a33cb in current term 1: Already voted for candidate fb45a4c0b9324c8f821b4e85142fc8ae in this term.
I20260925 04:36:06.227001 9544 leader_election.cc:302] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e1f54acfd3f4ac9bf9e9fefd10a33cb; no voters: c25cf5a5e65840d393e66b84603d3fbe, fb45a4c0b9324c8f821b4e85142fc8ae
I20260925 04:36:06.227317 9955 raft_consensus.cc:2726] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:36:06.229550 9462 catalog_manager.cc:5495] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae reported cstate change: term changed from 0 to 1, leader changed from <none> to fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195). New cstate: current_term: 1 leader_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } health_report { overall_health: UNKNOWN } } }
W20260925 04:36:06.252827 9659 tablet.cc:2369] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:06.292382 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.288885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.27.46.195:40643)
0925 04:36:06.289020 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.289024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.289038 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:36:06.289119 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.289122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.289182 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.289269 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:06.289607 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.290134 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.291043 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.291248 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.291325 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.291620 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:06.292045 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:36:06.292136 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.292196 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.292247 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:36:06.298668 9747 raft_consensus.cc:1261] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Refusing update from remote peer fb45a4c0b9324c8f821b4e85142fc8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:06.298708 9610 raft_consensus.cc:1261] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Refusing update from remote peer fb45a4c0b9324c8f821b4e85142fc8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:06.299367 9954 consensus_queue.cc:1035] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:06.299571 9959 consensus_queue.cc:1035] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:06.307101 9962 mvcc.cc:205] Tried to move back new op lower bound from 7333113717951049728 to 7333113717662552064. Current Snapshot: MvccSnapshot[applied={T|T < 7333113717951049728}]
I20260925 04:36:06.307888 9964 mvcc.cc:205] Tried to move back new op lower bound from 7333113717951049728 to 7333113717662552064. Current Snapshot: MvccSnapshot[applied={T|T < 7333113717951049728}]
I20260925 04:36:06.310818 9963 mvcc.cc:205] Tried to move back new op lower bound from 7333113717951049728 to 7333113717662552064. Current Snapshot: MvccSnapshot[applied={T|T < 7333113717951049728 or (T in {7333113717951049728})}]
I20260925 04:36:06.316051 9462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:06.316243 9462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:06.318719 9462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Sending DeleteTablet for 3 replicas of tablet 411600d4cde9442a8f74a58dedf3a670
I20260925 04:36:06.319597 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9526
I20260925 04:36:06.319584 9862 tablet_service.cc:1514] Processing DeleteTablet for tablet 411600d4cde9442a8f74a58dedf3a670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:06 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:41210
I20260925 04:36:06.319729 9590 tablet_service.cc:1514] Processing DeleteTablet for tablet 411600d4cde9442a8f74a58dedf3a670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:06 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:34310
I20260925 04:36:06.319726 9727 tablet_service.cc:1514] Processing DeleteTablet for tablet 411600d4cde9442a8f74a58dedf3a670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:06 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:44612
I20260925 04:36:06.320590 9973 tablet_replica.cc:330] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: stopping tablet replica
I20260925 04:36:06.320979 9973 raft_consensus.cc:2229] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:06.321321 9973 raft_consensus.cc:2258] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:06.322959 9973 ts_tablet_manager.cc:1905] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:06.325819 9972 tablet_replica.cc:330] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: stopping tablet replica
I20260925 04:36:06.326210 9972 raft_consensus.cc:2229] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:36:06.326602 9973 ts_tablet_manager.cc:1918] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:06.326717 9973 log.cc:1196] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/411600d4cde9442a8f74a58dedf3a670
I20260925 04:36:06.326843 9972 raft_consensus.cc:2258] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:06.327047 9973 ts_tablet_manager.cc:1939] T 411600d4cde9442a8f74a58dedf3a670 P c25cf5a5e65840d393e66b84603d3fbe: Deleting consensus metadata
I20260925 04:36:06.328446 9450 catalog_manager.cc:4841] TS c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097): tablet 411600d4cde9442a8f74a58dedf3a670 (table test-table [id=20db3788e58b46de85877eddccc33795]) successfully deleted
I20260925 04:36:06.328617 9972 ts_tablet_manager.cc:1905] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:36:06.329419 9448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:41927 (error 108)
I20260925 04:36:06.330540 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.330198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.0.0.1:34318)
0925 04:36:06.330338 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.330429 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:36:06.330629 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9663
I20260925 04:36:06.331655 9972 ts_tablet_manager.cc:1918] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:06.331749 9972 log.cc:1196] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/411600d4cde9442a8f74a58dedf3a670
I20260925 04:36:06.332099 9972 ts_tablet_manager.cc:1939] T 411600d4cde9442a8f74a58dedf3a670 P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting consensus metadata
W20260925 04:36:06.337783 9448 catalog_manager.cc:4584] TS 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 411600d4cde9442a8f74a58dedf3a670: Network error: Client connection negotiation failed: client connection to 127.27.46.193:41927: connect: Connection refused (error 111)
I20260925 04:36:06.338099 9448 catalog_manager.cc:4841] TS fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643): tablet 411600d4cde9442a8f74a58dedf3a670 (table test-table [id=20db3788e58b46de85877eddccc33795]) successfully deleted
I20260925 04:36:06.339608 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9798
I20260925 04:36:06.346405 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9432
I20260925 04:36:06.353335 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--webserver_interface=127.27.46.254
--webserver_port=37233
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:06.465389 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:06.465762 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:06.465863 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:06.470526 9975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:06.470634 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:06.470674 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:06.470712 9975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:06.470747 9975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:06.476159 9975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:06.477270 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:06.478312 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:06.492121 9982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:06.492158 9985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:06.492172 9975 server_base.cc:1034] running on GCE node
W20260925 04:36:06.492121 9983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:06.493144 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:06.494504 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:06.495680 9975 hybrid_clock.cc:648] HybridClock initialized: now 1790310966495636 us; error 58 us; skew 500 ppm
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:06.498600 9975 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:06.499503 9975 webserver.cc:460] Webserver started at http://127.27.46.254:37233/ using document root <none> and password file <none>
I20260925 04:36:06.500115 9975 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:06.500212 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:06.504321 9975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:36:06.506608 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:06.507915 9975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:36:06.508066 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "6c668e49f7db4bc7beefc7ddbef5004d"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "25367748b8c449558f7dd2d668be4e32"
server_key_iv: "a18e4c0cd9c64b3977415439a37d79fc"
server_key_version: "encryptionkey@0"
I20260925 04:36:06.508673 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:06.518265 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:06.521431 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:06.521672 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:06.529311 9975 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:46313
I20260925 04:36:06.529657 10044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:46313 every 8 connection(s)
I20260925 04:36:06.530622 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:06.535225 10045 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:36:06.537030 10045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap starting.
I20260925 04:36:06.540407 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 9975
I20260925 04:36:06.540917 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:41927
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=44309
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:06.555914 10045 log.cc:824] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:06.565475 10045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:06.565968 10045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap complete.
I20260925 04:36:06.570297 10045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:06.570721 10045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Initialized, Role: FOLLOWER
I20260925 04:36:06.571256 10045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:06.571386 10045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:06.571484 10045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:06.571607 10045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:36:06.573634 10045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:06.573987 10045 leader_election.cc:302] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c668e49f7db4bc7beefc7ddbef5004d; no voters:
I20260925 04:36:06.574364 10045 leader_election.cc:288] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:36:06.574451 10049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:36:06.574630 10049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 LEADER]: Becoming Leader. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Running, Role: LEADER
I20260925 04:36:06.574935 10049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:06.575664 10045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:06.575894 10049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c668e49f7db4bc7beefc7ddbef5004d. Latest consensus state: current_term: 2 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:06.576025 10049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:06.576391 10053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:06.576423 10050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:06.576643 10050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:06.581699 10053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:06.582413 10053 catalog_manager.cc:1259] Loaded cluster ID: 2e8a03a7a848432a9582759258b50ef0
I20260925 04:36:06.582495 10053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:06.583663 10053 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 04:36:06.586521 10053 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 04:36:06.591679 10053 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:06.592396 10053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Loaded TSK: 0
I20260925 04:36:06.593400 10053 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:36:06.678705 10047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:06.679039 10047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:06.679113 10047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:06.683388 10047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:06.683509 10047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:06.683606 10047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:36:06.689586 10047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:41927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=44309
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:06.690728 10047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:06.692026 10047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:06.702095 10075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:06.702255 10047 server_base.cc:1034] running on GCE node
W20260925 04:36:06.702080 10073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:06.702080 10072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:06.702836 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:06.703534 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:06.704761 10047 hybrid_clock.cc:648] HybridClock initialized: now 1790310966704743 us; error 38 us; skew 500 ppm
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:06.708729 10047 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:36:06.709846 10047 webserver.cc:460] Webserver started at http://127.27.46.193:44309/ using document root <none> and password file <none>
I20260925 04:36:06.710398 10047 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:06.710474 10047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:06.715003 10047 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:06.717366 10083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:06.718823 10047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:06.718930 10047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "90a7b4a3ba3ca1a16653c09c0e54b24d"
server_key_iv: "409ad6de8818a9fd6e496623ebd5dfb1"
server_key_version: "encryptionkey@0"
I20260925 04:36:06.719527 10047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:06.730548 10047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:06.734074 10047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:06.734300 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:06.735014 10047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:06.736320 10090 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:36:06.741803 10047 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:36:06.741912 10047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 04:36:06.742013 10047 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:36:06.743822 10047 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:36:06.743894 10047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 04:36:06.744228 10090 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap starting.
I20260925 04:36:06.754755 10047 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:41927
I20260925 04:36:06.754917 10198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:41927 every 8 connection(s)
I20260925 04:36:06.755920 10047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:36:06.758548 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10047
I20260925 04:36:06.759063 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:37097
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=39789
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:06.765139 10090 log.cc:824] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:06.774524 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.758139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:38627 (local address 127.27.46.254:46313)
0925 04:36:06.758746 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.758769 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.759552 (+ 783us) server_negotiation.cc:407] Connection header received
0925 04:36:06.761204 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.761224 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.761640 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.761991 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:06.763182 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.764178 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.766028 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.766670 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.766876 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.769415 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:06.769421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:06.769434 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:06.769463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:06.771249 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.771818 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.771828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.771832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.771953 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.772297 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.772300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.772302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.772658 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:06.772811 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.773421 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.773764 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":141,"threads_started":1}
I20260925 04:36:06.775543 10090 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:06.776018 10090 tablet_bootstrap.cc:492] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap complete.
I20260925 04:36:06.776151 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.758188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:38627)
0925 04:36:06.759095 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.759403 (+ 308us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.760728 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.762057 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.762066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.762515 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:06.762994 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.763009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.764343 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.764349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.765872 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.765888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.766833 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.766838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.767027 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.767728 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:06.767745 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:06.769199 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:06.771396 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.771402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.771415 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.771680 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.772055 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.772058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.772060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.772200 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.772780 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:06.772786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:06.773083 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.774949 (+ 1866us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.775259 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":156,"threads_started":1}
I20260925 04:36:06.777477 10090 ts_tablet_manager.cc:1397] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.011s
I20260925 04:36:06.778797 10199 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
I20260925 04:36:06.779057 10199 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:06.779574 10199 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:06.780520 10090 raft_consensus.cc:348] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.781064 10090 raft_consensus.cc:729] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, State: Initialized, Role: FOLLOWER
I20260925 04:36:06.781582 10090 consensus_queue.cc:260] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:06.782281 10009 ts_manager.cc:194] Registered new tserver with Master: 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:06.782702 10090 ts_tablet_manager.cc:1428] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:36:06.785840 10009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:38627
I20260925 04:36:06.788436 10199 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:36:06.796756 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.785364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.27.46.193:41927)
0925 04:36:06.785766 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.785778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.786262 (+ 484us) server_negotiation.cc:407] Connection header received
0925 04:36:06.786464 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.786469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.786671 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.786816 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:06.787612 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.788715 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.790057 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.790475 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.790563 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.793064 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:06.793072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:06.793082 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:06.793105 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:06.794825 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.795369 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.795373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.795375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.795454 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.795741 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.795743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.795744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.796005 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:06.796108 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.796483 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.796624 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
I20260925 04:36:06.796803 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.785280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.0.0.1:34324)
0925 04:36:06.786099 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.786144 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.786293 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.786893 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.786900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.787338 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:06.787468 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.787477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.788870 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.788874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.789827 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.789840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.790598 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.790601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.790710 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.791275 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:06.791291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:06.792918 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:06.794941 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.794951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.794956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.795253 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.795545 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.795548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.795550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.795651 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.796118 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:06.796124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:06.796261 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.796410 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.796614 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":126,"threads_started":1}
I20260925 04:36:06.797959 10133 tablet_service.cc:1514] Processing DeleteTablet for tablet 411600d4cde9442a8f74a58dedf3a670 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:06 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260925 04:36:06.798965 10208 tablet_replica.cc:330] stopping tablet replica
I20260925 04:36:06.799273 10208 raft_consensus.cc:2229] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:06.799537 10208 raft_consensus.cc:2258] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:06.800557 10208 ts_tablet_manager.cc:1905] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:06.804634 10208 ts_tablet_manager.cc:1918] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:06.804770 10208 log.cc:1196] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/411600d4cde9442a8f74a58dedf3a670
I20260925 04:36:06.805099 10208 ts_tablet_manager.cc:1939] T 411600d4cde9442a8f74a58dedf3a670 P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Deleting consensus metadata
I20260925 04:36:06.807034 9993 catalog_manager.cc:4841] TS 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927): tablet 411600d4cde9442a8f74a58dedf3a670 (table test-table [id=20db3788e58b46de85877eddccc33795]) successfully deleted
W20260925 04:36:06.887789 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:06.888144 10203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:06.888227 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:06.892465 10203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:06.892580 10203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:06.892683 10203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:36:06.898344 10203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:37097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=39789
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:06.899547 10203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:06.900731 10203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:06.911069 10216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:06.911100 10214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:06.911069 10213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:06.911355 10203 server_base.cc:1034] running on GCE node
I20260925 04:36:06.911895 10203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:06.912628 10203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:06.913789 10203 hybrid_clock.cc:648] HybridClock initialized: now 1790310966913767 us; error 34 us; skew 500 ppm
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:06.916682 10203 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:36:06.917677 10203 webserver.cc:460] Webserver started at http://127.27.46.194:39789/ using document root <none> and password file <none>
I20260925 04:36:06.918246 10203 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:06.918318 10203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:06.922425 10203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:36:06.924769 10224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:06.926198 10203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:36:06.926347 10203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c25cf5a5e65840d393e66b84603d3fbe"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "ed664f899c5afd25b8243207deffbf26"
server_key_iv: "e29a49d2f6d804cec00155ff9da46953"
server_key_version: "encryptionkey@0"
I20260925 04:36:06.926947 10203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:06.948736 10203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:06.952387 10203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:06.952618 10203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:06.953289 10203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:06.954185 10203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:06.954232 10203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:06.954313 10203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:06.954353 10203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:06.963615 10203 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37097
I20260925 04:36:06.963644 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37097 every 8 connection(s)
I20260925 04:36:06.964675 10203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:36:06.970129 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10203
I20260925 04:36:06.970577 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:40643
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=34473
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:36:06 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:06.980568 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.966620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:50585 (local address 127.27.46.254:46313)
0925 04:36:06.966845 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:06.966850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:06.967763 (+ 913us) server_negotiation.cc:407] Connection header received
0925 04:36:06.969139 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:06.969141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:06.969195 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:06.969271 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:06.970869 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.971740 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.973041 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.973319 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.973458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.976298 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:06.976302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:06.976304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:06.976333 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:06.978187 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.978768 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.978773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.978775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.978868 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:06.979280 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:06.979284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:06.979286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:06.979497 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:06.979617 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:06.980202 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 04:36:06.980358 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 04:36:06.981454 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:06.967081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:50585)
0925 04:36:06.967590 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:06.967636 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:06.968930 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:06.969475 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:06.969482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:06.969990 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:06.970595 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.970611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.971906 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.971909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.972877 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:06.972888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:06.973536 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:06.973541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:06.973734 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:06.974406 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:06.974424 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:06.976084 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:06.978328 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.978335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.978352 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.978642 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.978977 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:06.978980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:06.978982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:06.979163 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:06.979616 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:06.979626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:06.979918 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 04:36:06.980134 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 04:36:06.980506 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":102,"threads_started":1}
I20260925 04:36:06.982959 10339 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
I20260925 04:36:06.983234 10339 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:06.983786 10339 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:06.984995 10009 ts_manager.cc:194] Registered new tserver with Master: c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097)
I20260925 04:36:06.985814 10009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:50585
W20260925 04:36:07.091598 10342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:07.091897 10342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:07.091943 10342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:07.096195 10342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:07.096271 10342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:07.096341 10342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:36:07.102031 10342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:40643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=34473
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:07.103165 10342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:07.104257 10342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:07.114735 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.114781 10342 server_base.cc:1034] running on GCE node
W20260925 04:36:07.114720 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:07.114727 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.115468 10342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:07.116173 10342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:07.117347 10342 hybrid_clock.cc:648] HybridClock initialized: now 1790310967117328 us; error 30 us; skew 500 ppm
Sep 25 04:36:07 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:07.120066 10342 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:36:07.121043 10342 webserver.cc:460] Webserver started at http://127.27.46.195:34473/ using document root <none> and password file <none>
I20260925 04:36:07.121701 10342 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:07.121807 10342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:07.126037 10342 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:36:07.128237 10357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:07.129678 10342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:07.129817 10342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "fb45a4c0b9324c8f821b4e85142fc8ae"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "263709563e2e7d124dd14929a4c24404"
server_key_iv: "41c8157227478b085fb6b219f0c7ceee"
server_key_version: "encryptionkey@0"
I20260925 04:36:07.130452 10342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:07.157262 10342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:07.160460 10342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:07.160687 10342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:07.161348 10342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:07.162281 10342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:07.162371 10342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:07.162444 10342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:07.162488 10342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:07.172610 10342 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:40643
I20260925 04:36:07.172636 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:40643 every 8 connection(s)
I20260925 04:36:07.173790 10342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:36:07.178491 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10342
Sep 25 04:36:07 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:07.187597 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.179262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.27.46.254:46313)
0925 04:36:07.179893 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.179897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.179909 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:36:07.179950 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.179954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.179995 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.180083 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:07.180830 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.181469 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.182419 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.182670 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.182781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.184265 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:07.184268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:07.184270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:07.184295 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:07.185900 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:07.186329 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:07.186333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:07.186334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:07.186418 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:07.186695 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:07.186699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:07.186700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:07.186887 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:07.187101 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.187254 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.187380 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":549,"thread_start_us":76,"threads_started":1}
I20260925 04:36:07.189069 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.175839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:44529 (local address 127.27.46.254:46313)
0925 04:36:07.175982 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.175986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.176970 (+ 984us) server_negotiation.cc:407] Connection header received
0925 04:36:07.178542 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.178545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.178607 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.178690 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:07.180222 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.180761 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.182287 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.182531 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.182686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.185537 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:07.185540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:07.185543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:07.185587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:07.186931 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:07.187450 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:07.187455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:07.187456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:07.187536 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:07.187937 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:07.187940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:07.187941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:07.188124 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:07.188221 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.188812 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.188926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":53}
I20260925 04:36:07.190164 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.176296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:44529)
0925 04:36:07.176812 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.176858 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.178304 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.178875 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:07.178882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:07.179396 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:07.180098 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.180118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.180904 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.180907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.182149 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.182160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.182701 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.182705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.182897 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.183645 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:07.183662 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:07.185332 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:07.187057 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:07.187063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:07.187079 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:07.187342 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:07.187631 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:07.187635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:07.187637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:07.187839 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:07.188261 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:07.188268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:07.188569 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 04:36:07.188774 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 04:36:07.189151 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260925 04:36:07.191279 10472 heartbeater.cc:347] Connected to a master server at 127.27.46.254:46313
I20260925 04:36:07.191509 10472 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:07.191988 10472 heartbeater.cc:511] Master 127.27.46.254:46313 requested a full tablet report, sending...
I20260925 04:36:07.192976 10009 ts_manager.cc:194] Registered new tserver with Master: fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643)
I20260925 04:36:07.193617 10009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:44529
I20260925 04:36:07.201817 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:36:07.206386 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.202478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.27.46.254:46313)
0925 04:36:07.202616 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.202620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.202768 (+ 148us) server_negotiation.cc:407] Connection header received
0925 04:36:07.202916 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.202918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.203024 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.203113 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:07.203302 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.203814 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.204762 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.204999 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.205132 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.205431 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:07.206038 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:36:07.206136 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.206204 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.206257 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:36:07.207073 10009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:36:07.210414 10009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:36:07.218994 10133 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe29191b41642da98bab409ac543ecd (DEFAULT_TABLE table=test-table [id=287150179ded40318951561b7534c397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:07.219697 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe29191b41642da98bab409ac543ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:07.222574 10090 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap starting.
I20260925 04:36:07.223567 10090 tablet_bootstrap.cc:654] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:07.223976 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.218390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.0.0.1:44626)
0925 04:36:07.218896 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.218910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.218997 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.219782 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:07.219785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:07.219802 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:07.219886 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.219891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.221198 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.221205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.222337 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.222346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.223466 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.223470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.223570 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.223640 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 04:36:07.223711 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:36:07.223820 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260925 04:36:07.224159 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.217930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:40643 (local address 127.0.0.1:41224)
0925 04:36:07.218073 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.218089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.218173 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.219171 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:07.219177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:07.219199 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:07.219313 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.219319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.220972 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.220975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.222349 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.222361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.223687 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.223690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.223806 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.223841 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:36:07.223925 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:36:07.224007 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":58}
I20260925 04:36:07.224592 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.218848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.27.46.194:37097)
0925 04:36:07.219272 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.219284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.219306 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:36:07.219396 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.219404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.219621 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.219796 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:07.219974 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.221061 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.222468 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.223326 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.223469 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.223778 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.223836 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.223932 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":91,"threads_started":1}
I20260925 04:36:07.224862 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.218142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.27.46.195:40643)
0925 04:36:07.218571 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.218583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.218610 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:36:07.218696 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.218702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.218972 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.219189 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:07.219422 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.220824 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.222690 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.223532 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.224347 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.224518 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.224575 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.224692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260925 04:36:07.225119 10090 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: No bootstrap required, opened a new log
I20260925 04:36:07.225203 10090 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260925 04:36:07.225744 10090 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.225878 10090 raft_consensus.cc:374] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:07.225924 10090 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, State: Initialized, Role: FOLLOWER
I20260925 04:36:07.226047 10090 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.226311 10090 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:36:07.227844 10405 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe29191b41642da98bab409ac543ecd (DEFAULT_TABLE table=test-table [id=287150179ded40318951561b7534c397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:07.227939 10273 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe29191b41642da98bab409ac543ecd (DEFAULT_TABLE table=test-table [id=287150179ded40318951561b7534c397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:07.228942 10273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe29191b41642da98bab409ac543ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:07.228947 10405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe29191b41642da98bab409ac543ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:07.236053 10486 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Bootstrap starting.
I20260925 04:36:07.236049 10485 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Bootstrap starting.
I20260925 04:36:07.238845 10486 tablet_bootstrap.cc:654] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:07.238878 10485 tablet_bootstrap.cc:654] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:07.239732 10485 log.cc:824] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:07.239750 10486 log.cc:824] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:07.241727 10486 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: No bootstrap required, opened a new log
I20260925 04:36:07.241734 10485 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: No bootstrap required, opened a new log
I20260925 04:36:07.241940 10486 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:36:07.241946 10485 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:36:07.244951 10486 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.244951 10485 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.245222 10486 raft_consensus.cc:374] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:07.245240 10485 raft_consensus.cc:374] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:07.245292 10485 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb45a4c0b9324c8f821b4e85142fc8ae, State: Initialized, Role: FOLLOWER
I20260925 04:36:07.245286 10486 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25cf5a5e65840d393e66b84603d3fbe, State: Initialized, Role: FOLLOWER
I20260925 04:36:07.245803 10486 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.245803 10485 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.246528 10472 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
I20260925 04:36:07.246722 10339 heartbeater.cc:503] Master 127.27.46.254:46313 was elected leader, sending a full tablet report...
I20260925 04:36:07.246898 10486 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260925 04:36:07.247889 10485 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260925 04:36:07.257159 10200 tablet.cc:2369] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:07.363579 10489 raft_consensus.cc:482] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:07.363798 10489 raft_consensus.cc:504] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.365008 10489 leader_election.cc:288] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097), 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:07.370108 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.365405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:46133 (local address 127.27.46.194:37097)
0925 04:36:07.365568 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.365573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.365940 (+ 367us) server_negotiation.cc:407] Connection header received
0925 04:36:07.366003 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.366008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.366060 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.366131 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:07.366408 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.367195 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.368918 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.369486 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.369640 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.369705 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.369891 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.369981 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:36:07.370116 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.365405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:46947 (local address 127.27.46.193:41927)
0925 04:36:07.365739 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.365744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.365762 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:36:07.365823 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.365831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.365888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.366036 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:07.366416 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.367189 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.368918 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.369608 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.369709 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.369903 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.369948 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.370009 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260925 04:36:07.370110 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.365281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.27.46.195:46947)
0925 04:36:07.365404 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.365421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.365553 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.366073 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:07.366076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:07.366097 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:07.366288 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.366294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.367310 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.367313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.368784 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.368795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.369702 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.369705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.369790 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.369817 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:36:07.369872 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 04:36:07.369943 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":174}
I20260925 04:36:07.370117 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.365284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.27.46.195:46133)
0925 04:36:07.365837 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.365848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.365933 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.366145 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:07.366147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:07.366159 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:07.366288 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.366294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.367310 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.367313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.368784 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:07.368795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.369634 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.369636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:07.369748 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.369777 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:36:07.369857 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:36:07.369943 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":313,"thread_start_us":172,"threads_started":1}
I20260925 04:36:07.370981 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe" is_pre_election: true
I20260925 04:36:07.370967 10153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" is_pre_election: true
I20260925 04:36:07.371311 10293 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 0.
I20260925 04:36:07.371325 10153 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 0.
I20260925 04:36:07.371845 10360 leader_election.cc:302] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, c25cf5a5e65840d393e66b84603d3fbe, fb45a4c0b9324c8f821b4e85142fc8ae; no voters:
I20260925 04:36:07.372136 10489 raft_consensus.cc:2781] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:36:07.372237 10489 raft_consensus.cc:482] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:07.372318 10489 raft_consensus.cc:3037] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:07.373694 10489 raft_consensus.cc:504] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.374058 10489 leader_election.cc:288] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 election: Requested vote from peers c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097), 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:07.374437 10153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
I20260925 04:36:07.374511 10293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe"
I20260925 04:36:07.374567 10153 raft_consensus.cc:3037] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:07.374686 10293 raft_consensus.cc:3037] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:07.375653 10153 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 1.
I20260925 04:36:07.375725 10293 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb45a4c0b9324c8f821b4e85142fc8ae in term 1.
I20260925 04:36:07.376030 10360 leader_election.cc:302] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c25cf5a5e65840d393e66b84603d3fbe, fb45a4c0b9324c8f821b4e85142fc8ae; no voters:
I20260925 04:36:07.376230 10489 raft_consensus.cc:2781] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:07.376487 10489 raft_consensus.cc:686] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 LEADER]: Becoming Leader. State: Replica: fb45a4c0b9324c8f821b4e85142fc8ae, State: Running, Role: LEADER
I20260925 04:36:07.376917 10489 consensus_queue.cc:237] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.380367 10009 catalog_manager.cc:5495] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae reported cstate change: term changed from 0 to 1, leader changed from <none> to fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195). New cstate: current_term: 1 leader_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } health_report { overall_health: UNKNOWN } } }
I20260925 04:36:07.419092 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.415493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.27.46.195:40643)
0925 04:36:07.415647 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:07.415651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:07.415665 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:36:07.415743 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:07.415746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:07.415800 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:07.415897 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:07.416131 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.416629 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.417431 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:07.417663 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:07.417740 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:07.418093 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:07.418565 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:36:07.418669 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:07.418804 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 04:36:07.418873 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 04:36:07.424865 10473 tablet.cc:2369] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:07.425570 10293 raft_consensus.cc:1261] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Refusing update from remote peer fb45a4c0b9324c8f821b4e85142fc8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:07.425566 10153 raft_consensus.cc:1261] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Refusing update from remote peer fb45a4c0b9324c8f821b4e85142fc8ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:07.426326 10489 consensus_queue.cc:1035] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:07.426506 10494 consensus_queue.cc:1035] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:07.441435 10009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:07.441665 10009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:07.444247 10009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Sending DeleteTablet for 3 replicas of tablet 0fe29191b41642da98bab409ac543ecd
I20260925 04:36:07.445024 10133 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe29191b41642da98bab409ac543ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:07 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:34324
I20260925 04:36:07.445106 10273 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe29191b41642da98bab409ac543ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:07 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:44626
I20260925 04:36:07.445221 10405 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe29191b41642da98bab409ac543ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:07 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:41224
I20260925 04:36:07.445686 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10047
I20260925 04:36:07.445750 10505 tablet_replica.cc:330] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: stopping tablet replica
I20260925 04:36:07.446623 10507 tablet_replica.cc:330] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: stopping tablet replica
I20260925 04:36:07.447139 10507 raft_consensus.cc:2229] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:36:07.447642 10507 raft_consensus.cc:2258] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:07.449677 10507 ts_tablet_manager.cc:1905] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:07.453533 10507 ts_tablet_manager.cc:1918] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:07.453720 10507 log.cc:1196] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/0fe29191b41642da98bab409ac543ecd
I20260925 04:36:07.454198 10507 ts_tablet_manager.cc:1939] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Deleting consensus metadata
I20260925 04:36:07.455729 9993 catalog_manager.cc:4841] TS fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643): tablet 0fe29191b41642da98bab409ac543ecd (table test-table [id=287150179ded40318951561b7534c397]) successfully deleted
I20260925 04:36:07.455914 10506 tablet_replica.cc:330] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: stopping tablet replica
I20260925 04:36:07.456283 10506 raft_consensus.cc:2229] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:07.456646 10506 raft_consensus.cc:2258] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:36:07.457763 9993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:41927 (error 108)
I20260925 04:36:07.458067 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10203
I20260925 04:36:07.458359 10506 ts_tablet_manager.cc:1905] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:07.459213 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.458839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.0.0.1:34330)
0925 04:36:07.459045 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.459112 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260925 04:36:07.459447 9993 catalog_manager.cc:4584] TS 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fe29191b41642da98bab409ac543ecd: Network error: Client connection negotiation failed: client connection to 127.27.46.193:41927: connect: Connection refused (error 111)
I20260925 04:36:07.466859 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10342
I20260925 04:36:07.467489 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.467096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.0.0.1:44628)
0925 04:36:07.467323 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.467381 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 04:36:07.473701 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 9975
I20260925 04:36:07.481784 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--webserver_interface=127.27.46.254
--webserver_port=37233
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:07.594973 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:07.595250 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:07.595310 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:07.599416 10509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:07.599478 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:07.599515 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:07.599532 10509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:07.599592 10509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:07.605188 10509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:07.606359 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:07.607424 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:07.616000 10515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:07.615989 10514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.616091 10509 server_base.cc:1034] running on GCE node
W20260925 04:36:07.616003 10517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.616786 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:07.618024 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:07.619185 10509 hybrid_clock.cc:648] HybridClock initialized: now 1790310967619161 us; error 35 us; skew 500 ppm
Sep 25 04:36:07 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:07.621945 10509 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:07.622792 10509 webserver.cc:460] Webserver started at http://127.27.46.254:37233/ using document root <none> and password file <none>
I20260925 04:36:07.623296 10509 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:07.623359 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:07.627274 10509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:36:07.629487 10524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:07.630837 10509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:07.630964 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "6c668e49f7db4bc7beefc7ddbef5004d"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "25367748b8c449558f7dd2d668be4e32"
server_key_iv: "a18e4c0cd9c64b3977415439a37d79fc"
server_key_version: "encryptionkey@0"
I20260925 04:36:07.631603 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:07.653231 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:07.656659 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:07.656896 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:07.664213 10509 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:46313
I20260925 04:36:07.664503 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:46313 every 8 connection(s)
I20260925 04:36:07.665375 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:07.668869 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10509
I20260925 04:36:07.669315 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:41927
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=44309
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:07.669994 10578 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:36:07.672328 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap starting.
I20260925 04:36:07.697865 10578 log.cc:824] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:07.713683 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:07.714174 10578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d: Bootstrap complete.
I20260925 04:36:07.719971 10578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:07.720520 10578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Initialized, Role: FOLLOWER
I20260925 04:36:07.721099 10578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:07.721287 10578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:07.721364 10578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:07.721460 10578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:36:07.724193 10578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:07.724599 10578 leader_election.cc:302] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c668e49f7db4bc7beefc7ddbef5004d; no voters:
I20260925 04:36:07.724967 10578 leader_election.cc:288] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:36:07.725121 10582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:36:07.725415 10582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [term 3 LEADER]: Becoming Leader. State: Replica: 6c668e49f7db4bc7beefc7ddbef5004d, State: Running, Role: LEADER
I20260925 04:36:07.725764 10582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } }
I20260925 04:36:07.726332 10578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:07.727402 10584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c668e49f7db4bc7beefc7ddbef5004d. Latest consensus state: current_term: 3 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:07.727409 10583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c668e49f7db4bc7beefc7ddbef5004d" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46313 } } }
I20260925 04:36:07.727548 10583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:07.727548 10584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c668e49f7db4bc7beefc7ddbef5004d [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:07.729362 10591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:07.735281 10591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:07.735962 10591 catalog_manager.cc:1259] Loaded cluster ID: 2e8a03a7a848432a9582759258b50ef0
I20260925 04:36:07.736027 10591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 04:36:07.737145 10591 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f00418b49ad google::LogMessage::Fail() at ??:0
@ 0x7f00418b788a google::LogMessage::SendToLog() at ??:0
@ 0x7f00418b44de google::LogMessage::Flush() at ??:0
@ 0x7f00418b5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f004c1f9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f004c1fad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f004c1f7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f004c23b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0043b063d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0043b06c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0043b083f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0043af7512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f004513f6db start_thread at ??:0
@ 0x7f00403b261f clone at ??:0
W20260925 04:36:07.833943 10580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:07.834215 10580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:07.834261 10580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:07.838668 10580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:07.838757 10580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:07.838891 10580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:36:07.844488 10580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:41927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=44309
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:07.845695 10580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:07.846776 10580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:07.856750 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.856913 10580 server_base.cc:1034] running on GCE node
W20260925 04:36:07.856741 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:07.856799 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:07.857467 10580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:07.858295 10580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:07.859500 10580 hybrid_clock.cc:648] HybridClock initialized: now 1790310967859461 us; error 48 us; skew 500 ppm
Sep 25 04:36:07 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:07.863025 10580 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:36:07.864208 10580 webserver.cc:460] Webserver started at http://127.27.46.193:44309/ using document root <none> and password file <none>
I20260925 04:36:07.865008 10580 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:07.865108 10580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:07.870571 10580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:36:07.873459 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:07.875232 10580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:07.875392 10580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "90a7b4a3ba3ca1a16653c09c0e54b24d"
server_key_iv: "409ad6de8818a9fd6e496623ebd5dfb1"
server_key_version: "encryptionkey@0"
I20260925 04:36:07.876164 10580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:07.911686 10580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:07.915669 10580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:07.915913 10580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:07.916590 10580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:07.917873 10636 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:36:07.923477 10580 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:36:07.923592 10580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 04:36:07.923691 10580 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:36:07.925470 10580 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:36:07.925578 10580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:07.925899 10636 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap starting.
I20260925 04:36:07.935779 10580 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:41927
I20260925 04:36:07.935819 10744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:41927 every 8 connection(s)
I20260925 04:36:07.936853 10580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:36:07.938149 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10580
I20260925 04:36:07.938678 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:37097
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=39789
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:07.948936 10636 log.cc:824] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:07.958653 10636 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:07.959124 10636 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Bootstrap complete.
I20260925 04:36:07.960637 10636 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.012s
I20260925 04:36:07.964053 10636 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.964617 10636 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, State: Initialized, Role: FOLLOWER
I20260925 04:36:07.965154 10636 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:07.966367 10636 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260925 04:36:07.974532 10747 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:36:07.939195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:34045)
0925 04:36:07.939831 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.939894 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:07.941161 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:07.973773 (+ 32612us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":155,"threads_started":1}
W20260925 04:36:07.975047 10745 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46313 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:46313: Client connection negotiation failed: client connection to 127.27.46.254:46313: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:36:07.975886 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.975581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:41437)
0925 04:36:07.975761 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.975828 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:36:07.976570 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:07.976350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:58823)
0925 04:36:07.976465 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:07.976511 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:36:07.976765 10745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:36:08.058712 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:08.059073 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:08.059157 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:08.063239 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:08.063349 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:08.063470 10748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:36:08.069044 10748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:37097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=39789
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:08.070317 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:08.071449 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:08.081851 10756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:08.082032 10748 server_base.cc:1034] running on GCE node
W20260925 04:36:08.081851 10755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:08.081866 10758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:08.082592 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:08.083318 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:08.084513 10748 hybrid_clock.cc:648] HybridClock initialized: now 1790310968084490 us; error 35 us; skew 500 ppm
Sep 25 04:36:08 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:08.087445 10748 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:36:08.088395 10748 webserver.cc:460] Webserver started at http://127.27.46.194:39789/ using document root <none> and password file <none>
I20260925 04:36:08.089013 10748 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:08.089109 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:08.093251 10748 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:08.095621 10765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:08.096999 10748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:08.097143 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "c25cf5a5e65840d393e66b84603d3fbe"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "ed664f899c5afd25b8243207deffbf26"
server_key_iv: "e29a49d2f6d804cec00155ff9da46953"
server_key_version: "encryptionkey@0"
I20260925 04:36:08.097842 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:08.110677 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:08.113709 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:08.113938 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:08.114629 10748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:08.115792 10772 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:36:08.120254 10748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:36:08.120357 10748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 04:36:08.120453 10748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:36:08.122272 10748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:36:08.122344 10748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 04:36:08.122617 10772 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Bootstrap starting.
I20260925 04:36:08.129222 10772 log.cc:824] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:08.133109 10748 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:37097
I20260925 04:36:08.133160 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:37097 every 8 connection(s)
I20260925 04:36:08.134052 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:36:08.136298 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10748
I20260925 04:36:08.136792 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:40643
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:46313
--webserver_port=34473
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:08.137773 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.136157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:50325)
0925 04:36:08.136689 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.136823 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":157,"threads_started":1}
W20260925 04:36:08.138264 10882 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46313 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:46313: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
I20260925 04:36:08.138656 10772 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:08.138962 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.138686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:45321)
0925 04:36:08.138850 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.138906 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 04:36:08.139189 10772 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Bootstrap complete.
I20260925 04:36:08.139598 10772 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260925 04:36:08.139688 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.139418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:43063)
0925 04:36:08.139534 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.139578 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:36:08.140259 10882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:36:08.142738 10772 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:08.143188 10772 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25cf5a5e65840d393e66b84603d3fbe, State: Initialized, Role: FOLLOWER
I20260925 04:36:08.143669 10772 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:08.144747 10772 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:36:08.144883 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.144580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:44617)
0925 04:36:08.144727 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.144795 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 04:36:08.145610 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.145318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:32941)
0925 04:36:08.145457 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.145530 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 04:36:08.251807 10885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:08.252074 10885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:08.252120 10885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:08.256171 10885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:08.256233 10885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:08.256292 10885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:36:08.262086 10885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:40643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=34473
--tserver_master_addrs=127.27.46.254:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:08.263572 10885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:08.265043 10885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:08.276624 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:08.276657 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:08.276665 10885 server_base.cc:1034] running on GCE node
W20260925 04:36:08.276620 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:08.277172 10885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:08.277922 10885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:08.279116 10885 hybrid_clock.cc:648] HybridClock initialized: now 1790310968279101 us; error 26 us; skew 500 ppm
Sep 25 04:36:08 dist-test-slave-r8r3 krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:08.281932 10885 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:36:08.283017 10885 webserver.cc:460] Webserver started at http://127.27.46.195:34473/ using document root <none> and password file <none>
I20260925 04:36:08.283619 10885 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:08.283681 10885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:08.287629 10885 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:36:08.289924 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:08.291333 10885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:08.291482 10885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "fb45a4c0b9324c8f821b4e85142fc8ae"
format_stamp: "Formatted at 2026-09-25 04:36:05 on dist-test-slave-r8r3"
server_key: "263709563e2e7d124dd14929a4c24404"
server_key_iv: "41c8157227478b085fb6b219f0c7ceee"
server_key_version: "encryptionkey@0"
I20260925 04:36:08.292050 10885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:08.312767 10885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:08.316092 10885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:08.316344 10885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:08.317036 10885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:08.318030 10885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:08.318084 10885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:08.318163 10885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:08.318219 10885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:08.328217 10885 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:40643
I20260925 04:36:08.328231 11015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:40643 every 8 connection(s)
I20260925 04:36:08.329363 10885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:36:08.333370 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.331671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:36039)
0925 04:36:08.332199 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.332634 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
W20260925 04:36:08.333890 11016 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:46313 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:46313: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
I20260925 04:36:08.334570 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.334329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:39095)
0925 04:36:08.334455 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.334511 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:36:08.335260 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.335022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:33127)
0925 04:36:08.335154 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.335208 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 04:36:08.335407 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 10885
W20260925 04:36:08.335457 11016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:36:08.977849 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:08.977363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:34241)
0925 04:36:08.977676 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:08.977733 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":76,"threads_started":1}
I20260925 04:36:09.146736 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.146305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:41707)
0925 04:36:09.146570 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.146625 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260925 04:36:09.279975 11022 raft_consensus.cc:482] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:09.280164 11022 raft_consensus.cc:504] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:09.281339 11022 leader_election.cc:288] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643), c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097)
Sep 25 04:36:09 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.195@KRBTEST.COM
Sep 25 04:36:09 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.194@KRBTEST.COM
I20260925 04:36:09.295894 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.281623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:37097 (local address 127.27.46.193:59593)
0925 04:36:09.281794 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.281810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:09.281922 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:09.283566 (+ 1644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:09.283573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:09.284197 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:09.284705 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:09.284720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.286108 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.286118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:09.287102 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:09.287112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.287825 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:09.288067 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:09.288776 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:09.288792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:09.290875 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:09.293086 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:09.293091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:09.293102 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:09.293420 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:09.293875 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:09.293878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:09.293880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:09.294072 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:09.294724 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:09.294732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:09.295004 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 04:36:09.295216 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 04:36:09.295682 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":154,"spinlock_wait_cycles":5248}
I20260925 04:36:09.295940 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.281866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:59593 (local address 127.27.46.194:37097)
0925 04:36:09.282279 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:09.282293 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:09.282335 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:36:09.282673 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:09.282682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:09.283180 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:09.283467 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:09.284894 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.285947 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287235 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.287692 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287876 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:09.291048 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:09.291056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:09.291073 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:09.291110 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:09.292969 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:09.293579 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:09.293586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:09.293591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:09.293753 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:09.294206 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:09.294209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:09.294210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:09.294609 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:09.294725 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:09.295348 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 04:36:09.295719 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260925 04:36:09.296041 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.281880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:49709 (local address 127.27.46.195:40643)
0925 04:36:09.282326 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:09.282336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:09.282411 (+ 75us) server_negotiation.cc:407] Connection header received
0925 04:36:09.283246 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:09.283253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:09.283647 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:09.283937 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:09.284897 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.285958 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287229 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.287692 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287881 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:09.290770 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:09.290775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:09.290788 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:09.290817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:09.292969 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:09.294116 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:09.294121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:09.294127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:09.294255 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:09.294542 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:09.294545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:09.294547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:09.295011 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:09.295150 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:09.295546 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 04:36:09.295847 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":124,"threads_started":1}
I20260925 04:36:09.295946 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.281628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:40643 (local address 127.27.46.193:49709)
0925 04:36:09.282040 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.282053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:09.282160 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:09.283904 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:09.283907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:09.284199 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:09.284718 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:09.284727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.286113 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.286115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:09.287099 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:09.287109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:09.287825 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:09.287829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:09.288067 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:09.288828 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:09.288840 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:09.290523 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:09.293556 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:09.293560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:09.293562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:09.293740 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:09.294355 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:09.294359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:09.294361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:09.294443 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:09.295131 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:09.295133 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:09.295279 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 04:36:09.295380 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:36:09.295770 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":329,"spinlock_wait_cycles":3840,"thread_start_us":92,"threads_started":1}
I20260925 04:36:09.298369 10834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe" is_pre_election: true
I20260925 04:36:09.298736 10834 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e1f54acfd3f4ac9bf9e9fefd10a33cb in term 1.
W20260925 04:36:09.299023 10630 leader_election.cc:341] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643): Not found: Tablet not found: 0fe29191b41642da98bab409ac543ecd
I20260925 04:36:09.299506 10632 leader_election.cc:302] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, c25cf5a5e65840d393e66b84603d3fbe; no voters: fb45a4c0b9324c8f821b4e85142fc8ae
I20260925 04:36:09.299777 11022 raft_consensus.cc:2781] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:36:09.299867 11022 raft_consensus.cc:482] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:09.299921 11022 raft_consensus.cc:3037] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:36:09.302217 11022 raft_consensus.cc:504] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:09.302614 11022 leader_election.cc:288] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 election: Requested vote from peers fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643), c25cf5a5e65840d393e66b84603d3fbe (127.27.46.194:37097)
I20260925 04:36:09.303016 10834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe29191b41642da98bab409ac543ecd" candidate_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c25cf5a5e65840d393e66b84603d3fbe"
I20260925 04:36:09.303161 10834 raft_consensus.cc:3037] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 1 FOLLOWER]: Advancing to term 2
W20260925 04:36:09.303308 10630 leader_election.cc:341] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643): Not found: Tablet not found: 0fe29191b41642da98bab409ac543ecd
I20260925 04:36:09.305586 10834 raft_consensus.cc:2454] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e1f54acfd3f4ac9bf9e9fefd10a33cb in term 2.
I20260925 04:36:09.305881 10632 leader_election.cc:302] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, c25cf5a5e65840d393e66b84603d3fbe; no voters: fb45a4c0b9324c8f821b4e85142fc8ae
I20260925 04:36:09.306048 11022 raft_consensus.cc:2781] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:36:09.306250 11022 raft_consensus.cc:686] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [term 2 LEADER]: Becoming Leader. State: Replica: 9e1f54acfd3f4ac9bf9e9fefd10a33cb, State: Running, Role: LEADER
I20260925 04:36:09.306605 11022 consensus_queue.cc:237] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:09.306977 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.306707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:40393)
0925 04:36:09.306849 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.306910 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:36:09.336522 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.336100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:38679)
0925 04:36:09.336363 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.336424 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260925 04:36:09.749202 10630 consensus_peers.cc:487] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb -> Peer fb45a4c0b9324c8f821b4e85142fc8ae (127.27.46.195:40643): Couldn't send request to peer fb45a4c0b9324c8f821b4e85142fc8ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fe29191b41642da98bab409ac543ecd. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:36:09.876466 10834 raft_consensus.cc:1261] T 0fe29191b41642da98bab409ac543ecd P c25cf5a5e65840d393e66b84603d3fbe [term 2 FOLLOWER]: Refusing update from remote peer 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 04:36:09.877178 11022 consensus_queue.cc:1035] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 04:36:09.879401 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.879026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:60537)
0925 04:36:09.879251 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.879314 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260925 04:36:09.882090 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.881654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:45503)
0925 04:36:09.881881 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.881967 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260925 04:36:09.882417 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:09.882142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:50907)
0925 04:36:09.882291 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:09.882349 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:36:10.330878 11036 ts_tablet_manager.cc:927] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Initiating tablet copy from peer 9e1f54acfd3f4ac9bf9e9fefd10a33cb (127.27.46.193:41927)
I20260925 04:36:10.331566 11036 tablet_copy_client.cc:316] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: tablet copy: Beginning tablet copy session from remote peer at address 127.27.46.193:41927
I20260925 04:36:10.337790 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.337262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:37103)
0925 04:36:10.337620 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.337675 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":111,"threads_started":1}
Sep 25 04:36:10 dist-test-slave-r8r3 krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.193@KRBTEST.COM
I20260925 04:36:10.347025 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.332128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:41927 (local address 127.27.46.195:52843)
0925 04:36:10.332504 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.332527 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:10.332668 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:10.333131 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:10.333136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:10.333590 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:10.333710 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:10.333717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:10.334733 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:10.334736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:10.335402 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:10.335412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:10.336072 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:10.336075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:10.336163 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:10.340766 (+ 4603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:10.340781 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:10.342425 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:10.344872 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:10.344883 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:10.344892 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:10.345145 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:10.345472 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:10.345475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:10.345477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:10.345673 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:10.346265 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:10.346273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:10.346412 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 04:36:10.346613 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 04:36:10.346820 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":123,"threads_started":1}
I20260925 04:36:10.347062 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.332318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52843 (local address 127.27.46.193:41927)
0925 04:36:10.332690 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:10.332703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:10.332732 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:36:10.332794 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:10.332799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:10.332981 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:10.333113 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:10.333839 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:10.334583 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:10.335574 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:10.335945 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:10.336041 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:10.342608 (+ 6567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:10.342620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:10.342633 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:10.342662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:10.344750 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:10.345253 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:10.345259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:10.345260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:10.345359 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:10.345819 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:10.345823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:10.345824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:10.346122 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:10.346289 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:10.346730 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 04:36:10.346908 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260925 04:36:10.348323 10719 tablet_copy_service.cc:140] P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Received BeginTabletCopySession request for tablet 0fe29191b41642da98bab409ac543ecd from peer fb45a4c0b9324c8f821b4e85142fc8ae ({username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:52843)
I20260925 04:36:10.348541 10719 tablet_copy_service.cc:161] P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Beginning new tablet copy session on tablet 0fe29191b41642da98bab409ac543ecd from peer fb45a4c0b9324c8f821b4e85142fc8ae at {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:52843: session id = fb45a4c0b9324c8f821b4e85142fc8ae-0fe29191b41642da98bab409ac543ecd
I20260925 04:36:10.349836 10719 tablet_copy_source_session.cc:205] T 0fe29191b41642da98bab409ac543ecd P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:36:10.354000 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe29191b41642da98bab409ac543ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:10.360910 11036 tablet_copy_client.cc:791] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: tablet copy: Starting download of 0 data blocks...
I20260925 04:36:10.361150 11036 tablet_copy_client.cc:655] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: tablet copy: Starting download of 1 WAL segments...
I20260925 04:36:10.363108 11036 tablet_copy_client.cc:531] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:36:10.365031 11036 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Bootstrap starting.
I20260925 04:36:10.371039 11036 log.cc:824] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:10.378540 11036 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:10.378940 11036 tablet_bootstrap.cc:492] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Bootstrap complete.
I20260925 04:36:10.379298 11036 ts_tablet_manager.cc:1397] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent bootstrapping tablet: real 0.014s user 0.014s sys 0.000s
I20260925 04:36:10.381855 11036 raft_consensus.cc:348] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:10.382220 11036 raft_consensus.cc:729] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb45a4c0b9324c8f821b4e85142fc8ae, State: Initialized, Role: FOLLOWER
I20260925 04:36:10.382776 11036 consensus_queue.cc:260] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb45a4c0b9324c8f821b4e85142fc8ae" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 40643 } } peers { permanent_uuid: "c25cf5a5e65840d393e66b84603d3fbe" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 37097 } } peers { permanent_uuid: "9e1f54acfd3f4ac9bf9e9fefd10a33cb" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 41927 } }
I20260925 04:36:10.383966 11036 ts_tablet_manager.cc:1428] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:36:10.384173 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.383871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:59897)
0925 04:36:10.383995 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.384056 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 04:36:10.384732 10719 tablet_copy_service.cc:342] P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: Request end of tablet copy session fb45a4c0b9324c8f821b4e85142fc8ae-0fe29191b41642da98bab409ac543ecd received from {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:52843
I20260925 04:36:10.384865 10719 tablet_copy_service.cc:433] P 9e1f54acfd3f4ac9bf9e9fefd10a33cb: ending tablet copy session fb45a4c0b9324c8f821b4e85142fc8ae-0fe29191b41642da98bab409ac543ecd on tablet 0fe29191b41642da98bab409ac543ecd with peer fb45a4c0b9324c8f821b4e85142fc8ae
I20260925 04:36:10.384989 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.384715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:39415)
0925 04:36:10.384821 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.384907 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:36:10.754242 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.753856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:59445)
0925 04:36:10.754091 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.754147 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260925 04:36:10.754222 10970 raft_consensus.cc:1203] T 0fe29191b41642da98bab409ac543ecd P fb45a4c0b9324c8f821b4e85142fc8ae [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 04:36:10.755235 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.754985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:43457)
0925 04:36:10.755120 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.755170 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:36:10.883551 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:10.883128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:54575)
0925 04:36:10.883374 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:10.883448 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
I20260925 04:36:11.755369 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:11.754932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:60967)
0925 04:36:11.755177 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:11.755243 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260925 04:36:11.756356 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:11.755922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:42159)
0925 04:36:11.756206 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:11.756262 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":63,"threads_started":1}
I20260925 04:36:11.884681 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:11.884201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:36639)
0925 04:36:11.884512 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:11.884570 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":84,"threads_started":1}
I20260925 04:36:12.756682 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:12.756282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:38371)
0925 04:36:12.756531 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:12.756586 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260925 04:36:12.757596 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:12.757167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:42133)
0925 04:36:12.757381 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:12.757435 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260925 04:36:12.885871 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:12.885409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:49969)
0925 04:36:12.885715 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:12.885773 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":135,"threads_started":1}
I20260925 04:36:13.758267 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:13.757768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:59937)
0925 04:36:13.758084 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:13.758150 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":141,"threads_started":1}
I20260925 04:36:13.759087 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:13.758531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:44995)
0925 04:36:13.758789 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:13.758960 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260925 04:36:13.759443 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:13.759132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:37447)
0925 04:36:13.759343 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:13.759385 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260925 04:36:13.887174 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:13.886756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:47225)
0925 04:36:13.887013 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:13.887068 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260925 04:36:14.760208 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:14.759785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:38121)
0925 04:36:14.760066 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:14.760115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":120,"threads_started":1}
I20260925 04:36:14.760380 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:14.759992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:54575)
0925 04:36:14.760206 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:14.760265 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260925 04:36:14.888315 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:14.887901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:40435)
0925 04:36:14.888156 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:14.888209 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260925 04:36:15.761466 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:15.761025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:52001)
0925 04:36:15.761314 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:15.761372 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20260925 04:36:15.761596 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:15.761039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:45079)
0925 04:36:15.761399 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:15.761453 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
I20260925 04:36:15.889658 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:15.889056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:52599)
0925 04:36:15.889419 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:15.889480 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":85,"threads_started":1}
I20260925 04:36:16.762812 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:16.762354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:36619)
0925 04:36:16.762623 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:16.762675 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260925 04:36:16.762902 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:16.762333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:39557)
0925 04:36:16.762732 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:16.762807 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":97,"threads_started":1}
I20260925 04:36:16.890798 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:16.890390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:59005)
0925 04:36:16.890639 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:16.890698 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260925 04:36:17.764107 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:17.763664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:59151)
0925 04:36:17.763937 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:17.764008 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
I20260925 04:36:17.764143 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:17.763712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:43533)
0925 04:36:17.764001 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:17.764055 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260925 04:36:17.891958 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:17.891540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:36811)
0925 04:36:17.891799 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:17.891856 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260925 04:36:18.765573 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:18.765094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:38769)
0925 04:36:18.765368 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:18.765424 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260925 04:36:18.765582 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:18.765115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:51963)
0925 04:36:18.765392 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:18.765446 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
I20260925 04:36:18.893311 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:18.892874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:52749)
0925 04:36:18.893157 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:18.893210 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260925 04:36:19.766898 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:19.766427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:50575)
0925 04:36:19.766719 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:19.766778 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260925 04:36:19.766918 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:19.766427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:35261)
0925 04:36:19.766773 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:19.766826 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
I20260925 04:36:19.894482 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:19.894062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:39701)
0925 04:36:19.894311 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:19.894379 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260925 04:36:20.768265 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:20.767806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:48761)
0925 04:36:20.768095 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:20.768159 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260925 04:36:20.768347 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:20.767806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:50063)
0925 04:36:20.768178 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:20.768233 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
I20260925 04:36:20.895879 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:20.895375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:40523)
0925 04:36:20.895657 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:20.895723 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260925 04:36:21.769726 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:21.769219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:34633)
0925 04:36:21.769554 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:21.769621 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260925 04:36:21.769730 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:21.769235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:52761)
0925 04:36:21.769559 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:21.769629 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20260925 04:36:21.897073 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:21.896648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:59543)
0925 04:36:21.896916 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:21.896970 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260925 04:36:22.771167 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:22.770713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.195:35413)
0925 04:36:22.771020 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:22.771077 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260925 04:36:22.771191 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:22.770667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.193:56469)
0925 04:36:22.770987 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:22.771060 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
I20260925 04:36:22.898427 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:22.897918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:46313 (local address 127.27.46.194:59993)
0925 04:36:22.898237 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:22.898325 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":77,"threads_started":1}
I20260925 04:36:23.339985 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10580
I20260925 04:36:23.346915 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10748
I20260925 04:36:23.353753 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 10885
2026-09-25T04:36:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19461 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:36:23 dist-test-slave-r8r3 krb5kdc[11086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:36:23 dist-test-slave-r8r3 krb5kdc[11086](info): set up 2 sockets
Sep 25 04:36:23 dist-test-slave-r8r3 krb5kdc[11086](info): commencing operation
krb5kdc: starting...
W20260925 04:36:24.418135 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:36:24 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:36:24.543974 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:24.658689 11099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:24.659011 11099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:24.659137 11099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:24.663378 11099 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:36:24.663493 11099 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:36:24.663522 11099 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:36:24.663547 11099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:24.663607 11099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:24.663667 11099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:24.663707 11099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:24.663743 11099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:24.669468 11099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:24.670773 11099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:24.671916 11099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:36:24.681315 11099 server_base.cc:1034] running on GCE node
W20260925 04:36:24.681247 11107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:24.681247 11104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:24.681257 11105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:24.682301 11099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:24.683573 11099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:24.684795 11099 hybrid_clock.cc:648] HybridClock initialized: now 1790310984684779 us; error 42 us; skew 500 ppm
Sep 25 04:36:24 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:24.687767 11099 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:24.688714 11099 webserver.cc:460] Webserver started at http://127.27.46.254:36103/ using document root <none> and password file <none>
I20260925 04:36:24.689323 11099 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:24.689389 11099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:24.689658 11099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:24.691856 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "1d81920204db474690740df5de2544d0"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "149409c7f6e2514462f3b6ee5e69ecd1"
server_key_iv: "6a088b8a5c64cc8f1599bcaeaa3be148"
server_key_version: "encryptionkey@0"
I20260925 04:36:24.692476 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "1d81920204db474690740df5de2544d0"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "149409c7f6e2514462f3b6ee5e69ecd1"
server_key_iv: "6a088b8a5c64cc8f1599bcaeaa3be148"
server_key_version: "encryptionkey@0"
I20260925 04:36:24.696651 11099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:36:24.699499 11114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:24.700893 11099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:24.701059 11099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "1d81920204db474690740df5de2544d0"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "149409c7f6e2514462f3b6ee5e69ecd1"
server_key_iv: "6a088b8a5c64cc8f1599bcaeaa3be148"
server_key_version: "encryptionkey@0"
I20260925 04:36:24.701193 11099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:24.715282 11099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:24.718572 11099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:24.718806 11099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:24.726552 11099 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38215
I20260925 04:36:24.726879 11166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38215 every 8 connection(s)
I20260925 04:36:24.727691 11099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:24.730773 11167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:24.731143 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11099
I20260925 04:36:24.731287 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:36:24.731624 27835 external_mini_cluster.cc:1428] Setting key 3ebe23eddcc87b6e48d99cc47443c6fb
I20260925 04:36:24.737877 11167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap starting.
I20260925 04:36:24.740628 11167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Neither blocks nor log segments found. Creating new log.
Sep 25 04:36:24 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:24.741689 11167 log.cc:824] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:24.744227 11167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: No bootstrap required, opened a new log
I20260925 04:36:24.746579 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:24.733103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.27.46.254:38215)
0925 04:36:24.733654 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:24.733670 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:24.733709 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:36:24.734446 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:24.734465 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:24.734823 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:24.735196 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:24.736055 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:24.737061 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:24.738077 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:24.738579 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:24.738768 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:24.741685 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:24.741691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:24.741705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:24.741736 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:24.743590 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:24.744119 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:24.744124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:24.744129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:24.744241 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:24.744565 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:24.744567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:24.744569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:24.744970 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:24.745118 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:24.745443 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 04:36:24.745830 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":154,"threads_started":1}
I20260925 04:36:24.748226 11167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:24.748452 11167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:24.748514 11167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d81920204db474690740df5de2544d0, State: Initialized, Role: FOLLOWER
I20260925 04:36:24.749009 11167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:24.749145 11167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:24.749230 11167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:24.749356 11167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:24.750316 11167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:24.750680 11167 leader_election.cc:302] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d81920204db474690740df5de2544d0; no voters:
I20260925 04:36:24.751003 11167 leader_election.cc:288] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:36:24.751155 11172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:24.751410 11172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 LEADER]: Becoming Leader. State: Replica: 1d81920204db474690740df5de2544d0, State: Running, Role: LEADER
I20260925 04:36:24.751775 11172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:24.752343 11167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:24.753541 11173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:36:24.753649 11174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d81920204db474690740df5de2544d0. Latest consensus state: current_term: 1 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:36:24.753788 11174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:24.754072 11173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:24.754206 11181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:24.758141 11181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:24.764946 11181 catalog_manager.cc:1347] Generated new cluster ID: 3fe1ed79987a44e58a295492c86f6ce4
I20260925 04:36:24.765051 11181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:24.778738 11181 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:36:24.779744 11181 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:24.787642 11181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Generated new TSK 0
I20260925 04:36:24.788435 11181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:36:24.830904 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:24.961388 11193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:24.961817 11193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:24.961865 11193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:24.966121 11193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:24.966202 11193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:24.966264 11193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:36:24.972031 11193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:24.973258 11193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:24.974411 11193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:24.985252 11198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:24.985236 11199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:24.985354 11193 server_base.cc:1034] running on GCE node
W20260925 04:36:24.985239 11201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:24.986341 11193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:24.987089 11193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:24.988257 11193 hybrid_clock.cc:648] HybridClock initialized: now 1790310984988231 us; error 37 us; skew 500 ppm
Sep 25 04:36:24 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:24.991516 11193 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:36:24.992581 11193 webserver.cc:460] Webserver started at http://127.27.46.193:40381/ using document root <none> and password file <none>
I20260925 04:36:24.993229 11193 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:24.993297 11193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:24.993611 11193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:24.995936 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "2209b5979dc24803ada11e45a668a61b"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "86ffcbc5ce0e398c0801c1f8bf9eafb3"
server_key_iv: "6a9c3e3b53aab6c50de07cfcc1aea810"
server_key_version: "encryptionkey@0"
I20260925 04:36:24.996606 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "2209b5979dc24803ada11e45a668a61b"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "86ffcbc5ce0e398c0801c1f8bf9eafb3"
server_key_iv: "6a9c3e3b53aab6c50de07cfcc1aea810"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.001149 11193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:36:25.004020 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.005581 11193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:36:25.005728 11193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "2209b5979dc24803ada11e45a668a61b"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "86ffcbc5ce0e398c0801c1f8bf9eafb3"
server_key_iv: "6a9c3e3b53aab6c50de07cfcc1aea810"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.005875 11193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:25.027590 11193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:25.030925 11193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:25.031177 11193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:25.031847 11193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:25.032775 11193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:25.032855 11193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.032930 11193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:25.032984 11193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.042263 11193 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:35007
I20260925 04:36:25.042290 11321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:35007 every 8 connection(s)
I20260925 04:36:25.043248 11193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:36:25.049623 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11193
I20260925 04:36:25.049768 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:36:25.050127 27835 external_mini_cluster.cc:1428] Setting key acd5e1efe42413a6222bebd295b48599
Sep 25 04:36:25 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:25.059669 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.045243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48061 (local address 127.27.46.254:38215)
0925 04:36:25.045399 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.045403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.046613 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 04:36:25.047598 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.047601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.047667 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.047783 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:25.049102 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.049757 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.051262 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.051607 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.051722 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.055099 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:25.055106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:25.055112 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:25.055152 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:25.057037 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.057706 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.057712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.057716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.057824 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.058211 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.058218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.058223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.058535 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:25.058653 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.059241 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.059437 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:36:25.060453 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.045724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:48061)
0925 04:36:25.046444 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.046490 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.047363 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.047993 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.048001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.048399 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:25.048930 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.048945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.049967 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.049971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.051132 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.051144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.051801 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.051805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.052063 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.052910 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:25.052939 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:25.054860 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:25.057201 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.057206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.057224 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.057594 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.057937 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.057939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.057940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.058090 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.058657 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:25.058663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:25.058951 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.059121 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.059427 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:36:25.061962 11322 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:25.062229 11322 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:25.062799 11322 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:36:25.064564 11131 ts_manager.cc:194] Registered new tserver with Master: 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:25.066277 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:48061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:36:25.080533 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:25.200266 11327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:25.200568 11327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:25.200660 11327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:25.204988 11327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:25.205061 11327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:25.205173 11327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:36:25.211117 11327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:25.212350 11327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:25.213461 11327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:25.224030 11332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:25.224032 11335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:25.224032 11333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:25.224440 11327 server_base.cc:1034] running on GCE node
I20260925 04:36:25.224954 11327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:25.225718 11327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:25.226881 11327 hybrid_clock.cc:648] HybridClock initialized: now 1790310985226862 us; error 32 us; skew 500 ppm
Sep 25 04:36:25 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:25.229813 11327 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:36:25.230813 11327 webserver.cc:460] Webserver started at http://127.27.46.194:41463/ using document root <none> and password file <none>
I20260925 04:36:25.231400 11327 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:25.231491 11327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:25.231740 11327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:25.233959 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "f446dda56bee8bd6592342186ff0a78a"
server_key_iv: "a4f13f7341173bb3bf045f5727776e42"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.234535 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "f446dda56bee8bd6592342186ff0a78a"
server_key_iv: "a4f13f7341173bb3bf045f5727776e42"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.238448 11327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:36:25.241096 11342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.242444 11327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:36:25.242579 11327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "f446dda56bee8bd6592342186ff0a78a"
server_key_iv: "a4f13f7341173bb3bf045f5727776e42"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.242702 11327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:25.262290 11327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:25.266008 11327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:25.266232 11327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:25.266898 11327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:25.267845 11327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:25.267910 11327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.267984 11327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:25.268023 11327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.277349 11327 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:38071
I20260925 04:36:25.277366 11457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:38071 every 8 connection(s)
I20260925 04:36:25.278388 11327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
Sep 25 04:36:25 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:25.288371 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11327
I20260925 04:36:25.288511 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:36:25.288825 27835 external_mini_cluster.cc:1428] Setting key de6cf78f41c4a1fc7309683245da8da0
I20260925 04:36:25.292568 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.280254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:38711 (local address 127.27.46.254:38215)
0925 04:36:25.280412 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.280416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.281622 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 04:36:25.282549 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.282552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.282610 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.282693 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:25.283994 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.284535 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.285787 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.286012 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.286098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.288699 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:25.288702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:25.288704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:25.288731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:25.290358 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.290922 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.290925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.290927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.291013 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.291369 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.291373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.291375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.291598 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:25.291708 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.292270 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.292401 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:36:25.293541 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.280685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:38711)
0925 04:36:25.281379 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.281424 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.282331 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.282882 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.282888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.283262 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:25.283798 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.283812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.284666 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.284669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.285674 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.285684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.286164 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.286167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.286311 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.286943 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:25.286961 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:25.288457 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:25.290518 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.290525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.290538 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.290800 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.291111 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.291114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.291116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.291271 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.291709 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:25.291718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:25.291992 (+ 274us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.292217 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.292589 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":128,"threads_started":1}
I20260925 04:36:25.294865 11458 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:25.295151 11458 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:25.295862 11458 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
I20260925 04:36:25.297075 11131 ts_manager.cc:194] Registered new tserver with Master: d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
I20260925 04:36:25.297724 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:38711
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:36:25.316352 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:25.434315 11463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:25.434602 11463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:25.434690 11463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:25.438861 11463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:25.438944 11463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:25.439042 11463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:36:25.444649 11463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:25.445983 11463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:25.447150 11463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:36:25.457569 11463 server_base.cc:1034] running on GCE node
W20260925 04:36:25.457515 11469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:25.457533 11470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:25.457517 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:25.458231 11463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:25.458921 11463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:25.460140 11463 hybrid_clock.cc:648] HybridClock initialized: now 1790310985460121 us; error 31 us; skew 500 ppm
Sep 25 04:36:25 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:25.463080 11463 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:36:25.464023 11463 webserver.cc:460] Webserver started at http://127.27.46.195:33457/ using document root <none> and password file <none>
I20260925 04:36:25.464578 11463 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:25.464638 11463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:25.464811 11463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:36:25.467003 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "83947d33083d403395a91ab4db50945d"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "0e3e4c6edf7b332f77b9473b4034ada0"
server_key_iv: "4e533259a4c98a47398a2bda360a13ed"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.467559 11463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "83947d33083d403395a91ab4db50945d"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "0e3e4c6edf7b332f77b9473b4034ada0"
server_key_iv: "4e533259a4c98a47398a2bda360a13ed"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.471591 11463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:36:25.474211 11479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.475625 11463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:36:25.475798 11463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "83947d33083d403395a91ab4db50945d"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "0e3e4c6edf7b332f77b9473b4034ada0"
server_key_iv: "4e533259a4c98a47398a2bda360a13ed"
server_key_version: "encryptionkey@0"
I20260925 04:36:25.475932 11463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:25.494684 11463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:25.498180 11463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:25.498418 11463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:25.499073 11463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:25.500038 11463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:25.500088 11463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.500154 11463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:25.500186 11463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:25.510115 11463 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:39221
I20260925 04:36:25.510145 11594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:39221 every 8 connection(s)
I20260925 04:36:25.511183 11463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:36:25.513604 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11463
I20260925 04:36:25.513741 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:36:25.514108 27835 external_mini_cluster.cc:1428] Setting key 24146644f55119055d936d116a1e878a
Sep 25 04:36:25 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790310985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:25.525112 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.513334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:54079 (local address 127.27.46.254:38215)
0925 04:36:25.513901 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.513907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.514769 (+ 862us) server_negotiation.cc:407] Connection header received
0925 04:36:25.515733 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.515736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.515782 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.515853 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:25.517111 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.517717 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.518856 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.519064 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.519141 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.521669 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:25.521673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:25.521675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:25.521703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:25.522979 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.523533 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.523536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.523537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.523639 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.523973 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.523975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.523977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.524206 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:25.524295 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.524834 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.524970 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260925 04:36:25.526057 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.513853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:54079)
0925 04:36:25.514598 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.514655 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.515497 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.516042 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.516051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.516478 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:25.516978 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.516992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.517835 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.517838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.518744 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.518754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.519197 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.519199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.519350 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.519914 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:25.519932 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:25.521435 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:25.523088 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.523093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.523107 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.523392 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.523742 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:25.523744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:25.523745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:25.523883 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:25.524301 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:25.524306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:25.524568 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.524736 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.525016 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":126,"threads_started":1}
I20260925 04:36:25.527436 11595 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:25.527704 11595 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:25.528239 11595 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
I20260925 04:36:25.529323 11131 ts_manager.cc:194] Registered new tserver with Master: 83947d33083d403395a91ab4db50945d (127.27.46.195:39221)
I20260925 04:36:25.529963 11131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:54079
I20260925 04:36:25.540532 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:36:25.550072 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.542687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.27.46.254:38215)
0925 04:36:25.542844 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.542849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.543016 (+ 167us) server_negotiation.cc:407] Connection header received
0925 04:36:25.543125 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.543127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.543173 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.543263 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:25.543954 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.544491 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.545319 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.545596 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.545675 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.547082 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:25.547085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:25.547087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:25.547112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:25.548513 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.548899 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.548901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.548903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.548974 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:25.549194 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:25.549196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:25.549197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:25.549446 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:25.549613 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.549812 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.549931 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:36:25.553804 11131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:36:25.557077 11131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:36:25.572948 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.566339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39221 (local address 127.0.0.1:52196)
0925 04:36:25.567189 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.567243 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.567692 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.568647 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.568650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.568676 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:25.568813 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.568820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.570237 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.570242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.571356 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.571368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572384 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.572481 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572508 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.572638 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.572789 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":47,"thread_start_us":90,"threads_started":1}
I20260925 04:36:25.572962 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.566457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.27.46.195:39221)
0925 04:36:25.567147 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.567161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.567398 (+ 237us) server_negotiation.cc:407] Connection header received
0925 04:36:25.567872 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.567878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.568088 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.568209 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:25.568965 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.570110 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.571507 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572112 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572182 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572301 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.572677 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.572788 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":190,"threads_started":1}
I20260925 04:36:25.573444 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.567227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.27.46.193:35007)
0925 04:36:25.567606 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.567613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.568004 (+ 391us) server_negotiation.cc:407] Connection header received
0925 04:36:25.568088 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.568093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.568259 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.568397 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:25.568718 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.569932 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.571517 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572337 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572444 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572582 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.573185 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.573283 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
I20260925 04:36:25.573602 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.566841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.27.46.194:38071)
0925 04:36:25.567259 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.567270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.567535 (+ 265us) server_negotiation.cc:407] Connection header received
0925 04:36:25.567845 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.567852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.568058 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.568162 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:25.568750 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.569651 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.571510 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572337 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572442 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572582 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.573024 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.573135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":126,"threads_started":1}
I20260925 04:36:25.573848 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.566988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.0.0.1:59782)
0925 04:36:25.567781 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.567795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.567880 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.568383 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.568385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.568401 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:25.568572 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.568579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.570055 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.570059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.571383 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.571395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572653 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.572840 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572881 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.572958 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.573039 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":714,"thread_start_us":102,"threads_started":1}
I20260925 04:36:25.574334 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.566420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38071 (local address 127.0.0.1:47870)
0925 04:36:25.567421 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.567437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.567646 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.568370 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.568376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.568416 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:25.568572 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.568579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.569782 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.569787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.571340 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.571353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.572669 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.572672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.572836 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.572883 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.574115 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.574206 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":124,"spinlock_wait_cycles":11392,"thread_start_us":103,"threads_started":1}
I20260925 04:36:25.576357 11256 tablet_service.cc:1467] Processing CreateTablet for tablet d9d21325a437442c91f5ffbd2238209f (DEFAULT_TABLE table=test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:25.576359 11528 tablet_service.cc:1467] Processing CreateTablet for tablet d9d21325a437442c91f5ffbd2238209f (DEFAULT_TABLE table=test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:25.577445 11528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d21325a437442c91f5ffbd2238209f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:25.577447 11256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d21325a437442c91f5ffbd2238209f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:25.579816 11391 tablet_service.cc:1467] Processing CreateTablet for tablet d9d21325a437442c91f5ffbd2238209f (DEFAULT_TABLE table=test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:25.581009 11391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d21325a437442c91f5ffbd2238209f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:25.584800 11612 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Bootstrap starting.
I20260925 04:36:25.585533 11613 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Bootstrap starting.
I20260925 04:36:25.587594 11612 tablet_bootstrap.cc:654] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:25.587592 11613 tablet_bootstrap.cc:654] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:25.588591 11614 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Bootstrap starting.
I20260925 04:36:25.588590 11613 log.cc:824] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:25.589435 11612 log.cc:824] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:25.591130 11613 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: No bootstrap required, opened a new log
I20260925 04:36:25.591215 11612 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: No bootstrap required, opened a new log
I20260925 04:36:25.591372 11613 ts_tablet_manager.cc:1397] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:36:25.591420 11612 ts_tablet_manager.cc:1397] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:36:25.591672 11614 tablet_bootstrap.cc:654] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:25.592551 11614 log.cc:824] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:25.594811 11614 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: No bootstrap required, opened a new log
I20260925 04:36:25.595058 11614 ts_tablet_manager.cc:1397] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 04:36:25.594839 11612 raft_consensus.cc:348] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.595192 11612 raft_consensus.cc:374] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:25.595249 11612 raft_consensus.cc:729] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83947d33083d403395a91ab4db50945d, State: Initialized, Role: FOLLOWER
I20260925 04:36:25.595737 11612 consensus_queue.cc:260] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.596168 11613 raft_consensus.cc:348] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.596443 11613 raft_consensus.cc:374] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:25.596493 11613 raft_consensus.cc:729] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2209b5979dc24803ada11e45a668a61b, State: Initialized, Role: FOLLOWER
I20260925 04:36:25.596917 11613 consensus_queue.cc:260] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.597011 11612 ts_tablet_manager.cc:1428] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:36:25.597391 11595 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
I20260925 04:36:25.597935 11613 ts_tablet_manager.cc:1428] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260925 04:36:25.598307 11322 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
I20260925 04:36:25.599071 11614 raft_consensus.cc:348] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.599349 11614 raft_consensus.cc:374] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:25.599514 11614 raft_consensus.cc:729] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92bd3ecf5e24096bdf409fb6cd794af, State: Initialized, Role: FOLLOWER
I20260925 04:36:25.600040 11614 consensus_queue.cc:260] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.600759 11458 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
I20260925 04:36:25.601162 11614 ts_tablet_manager.cc:1428] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260925 04:36:25.762283 11596 tablet.cc:2369] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:36:25.779807 11459 tablet.cc:2369] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:36:25.794756 11323 tablet.cc:2369] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:25.841464 11618 raft_consensus.cc:482] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:25.841804 11618 raft_consensus.cc:504] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.842998 11618 leader_election.cc:288] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071), 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:25.847617 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.843476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:39681 (local address 127.27.46.194:38071)
0925 04:36:25.843606 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.843610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.843997 (+ 387us) server_negotiation.cc:407] Connection header received
0925 04:36:25.844144 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.844147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.844236 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.844324 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:25.844558 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.845217 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.846605 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.847000 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.847096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.847140 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.847391 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.847482 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:36:25.847620 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.843323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38071 (local address 127.27.46.195:39681)
0925 04:36:25.843880 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.843895 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.844010 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.844340 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.844342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.844358 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:25.844447 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.844451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.845303 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.845306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.846453 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.846460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.847117 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.847121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.847233 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.847260 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.847347 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.847427 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":189,"spinlock_wait_cycles":1792,"thread_start_us":124,"threads_started":1}
I20260925 04:36:25.847757 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.843492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:44017 (local address 127.27.46.193:35007)
0925 04:36:25.843648 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.843654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.843674 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:36:25.843738 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.843741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.843803 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.843872 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:25.844228 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.845020 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.846589 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.847132 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.847246 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.847292 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.847551 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.847623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:36:25.848155 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.843323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.27.46.195:44017)
0925 04:36:25.843514 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.843532 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:25.843645 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:25.843923 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:25.843930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:25.843956 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:25.844095 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.844102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.845130 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.845132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.846429 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:25.846440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.847263 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.847266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:25.847413 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.847436 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:36:25.847513 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:36:25.847581 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":25}
I20260925 04:36:25.848300 11412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d21325a437442c91f5ffbd2238209f" candidate_uuid: "83947d33083d403395a91ab4db50945d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" is_pre_election: true
I20260925 04:36:25.848680 11412 raft_consensus.cc:2454] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83947d33083d403395a91ab4db50945d in term 0.
I20260925 04:36:25.848850 11276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d21325a437442c91f5ffbd2238209f" candidate_uuid: "83947d33083d403395a91ab4db50945d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2209b5979dc24803ada11e45a668a61b" is_pre_election: true
I20260925 04:36:25.849169 11276 raft_consensus.cc:2454] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83947d33083d403395a91ab4db50945d in term 0.
I20260925 04:36:25.849242 11481 leader_election.cc:302] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af; no voters:
I20260925 04:36:25.849617 11618 raft_consensus.cc:2781] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:36:25.849731 11618 raft_consensus.cc:482] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:25.849793 11618 raft_consensus.cc:3037] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:25.850883 11618 raft_consensus.cc:504] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.851290 11618 leader_election.cc:288] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [CANDIDATE]: Term 1 election: Requested vote from peers d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071), 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:25.851749 11412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d21325a437442c91f5ffbd2238209f" candidate_uuid: "83947d33083d403395a91ab4db50945d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
I20260925 04:36:25.851877 11276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d21325a437442c91f5ffbd2238209f" candidate_uuid: "83947d33083d403395a91ab4db50945d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2209b5979dc24803ada11e45a668a61b"
I20260925 04:36:25.851910 11412 raft_consensus.cc:3037] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:25.852046 11276 raft_consensus.cc:3037] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:25.853124 11412 raft_consensus.cc:2454] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83947d33083d403395a91ab4db50945d in term 1.
I20260925 04:36:25.853165 11276 raft_consensus.cc:2454] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83947d33083d403395a91ab4db50945d in term 1.
I20260925 04:36:25.853539 11481 leader_election.cc:302] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af; no voters:
I20260925 04:36:25.853744 11618 raft_consensus.cc:2781] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:25.854017 11618 raft_consensus.cc:686] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 1 LEADER]: Becoming Leader. State: Replica: 83947d33083d403395a91ab4db50945d, State: Running, Role: LEADER
I20260925 04:36:25.854420 11618 consensus_queue.cc:237] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:25.857981 11129 catalog_manager.cc:5495] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83947d33083d403395a91ab4db50945d (127.27.46.195). New cstate: current_term: 1 leader_uuid: "83947d33083d403395a91ab4db50945d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } health_report { overall_health: UNKNOWN } } }
I20260925 04:36:25.939152 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.935560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.27.46.195:39221)
0925 04:36:25.935697 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:25.935701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:25.935717 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:36:25.935801 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:25.935803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:25.935863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:25.935970 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:25.936171 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.936679 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.937561 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:25.937777 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:25.937859 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:25.938168 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:25.938734 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:36:25.938867 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:25.938941 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 04:36:25.939005 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:36:25.945986 11412 raft_consensus.cc:1261] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Refusing update from remote peer 83947d33083d403395a91ab4db50945d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:25.945976 11276 raft_consensus.cc:1261] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Refusing update from remote peer 83947d33083d403395a91ab4db50945d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:25.946904 11622 consensus_queue.cc:1035] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:25.947151 11618 consensus_queue.cc:1035] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:25.958616 11626 mvcc.cc:205] Tried to move back new op lower bound from 7333113798425993216 to 7333113798062387200. Current Snapshot: MvccSnapshot[applied={T|T < 7333113798425993216}]
I20260925 04:36:25.964347 11129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:25.964531 11129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:25.967475 11129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Sending DeleteTablet for 3 replicas of tablet d9d21325a437442c91f5ffbd2238209f
I20260925 04:36:25.968577 11391 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d21325a437442c91f5ffbd2238209f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47870
I20260925 04:36:25.968664 11256 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d21325a437442c91f5ffbd2238209f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:59782
I20260925 04:36:25.968672 11528 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d21325a437442c91f5ffbd2238209f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:52196
I20260925 04:36:25.969218 11636 tablet_replica.cc:330] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: stopping tablet replica
I20260925 04:36:25.969439 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11193
I20260925 04:36:25.969482 11635 tablet_replica.cc:330] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: stopping tablet replica
I20260925 04:36:25.969631 11636 raft_consensus.cc:2229] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:36:25.970077 11636 raft_consensus.cc:2258] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:25.970333 11637 tablet_replica.cc:330] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: stopping tablet replica
I20260925 04:36:25.970642 11637 raft_consensus.cc:2229] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:25.970992 11637 raft_consensus.cc:2258] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:25.972847 11636 ts_tablet_manager.cc:1905] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:25.975914 11636 ts_tablet_manager.cc:1918] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:25.976032 11636 log.cc:1196] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/d9d21325a437442c91f5ffbd2238209f
I20260925 04:36:25.976368 11636 ts_tablet_manager.cc:1939] T d9d21325a437442c91f5ffbd2238209f P 83947d33083d403395a91ab4db50945d: Deleting consensus metadata
I20260925 04:36:25.977851 11637 ts_tablet_manager.cc:1905] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:25.977835 11117 catalog_manager.cc:4841] TS 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): tablet d9d21325a437442c91f5ffbd2238209f (table test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]) successfully deleted
W20260925 04:36:25.977931 11115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:35007 (error 108)
I20260925 04:36:25.979034 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.978703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.0.0.1:59792)
0925 04:36:25.978854 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.978939 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:36:25.979161 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11327
W20260925 04:36:25.980785 11115 catalog_manager.cc:4584] TS 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9d21325a437442c91f5ffbd2238209f: Network error: Client connection negotiation failed: client connection to 127.27.46.193:35007: connect: Connection refused (error 111)
I20260925 04:36:25.986280 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11463
I20260925 04:36:25.986537 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:25.986217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38071 (local address 127.0.0.1:47886)
0925 04:36:25.986359 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:25.986428 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:38071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 04:36:25.993094 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11099
I20260925 04:36:46.000586 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--webserver_interface=127.27.46.254
--webserver_port=36103
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:36:46.108095 11639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:46.108363 11639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:46.108438 11639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:46.112182 11639 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:36:46.112255 11639 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:36:46.112283 11639 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:36:46.112317 11639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:36:46.112355 11639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:46.112416 11639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:46.112453 11639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:36:46.112486 11639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:36:46.117318 11639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=36103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:46.118280 11639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:46.119246 11639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:46.132433 11645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:46.132508 11639 server_base.cc:1034] running on GCE node
W20260925 04:36:46.132467 11648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.132403 11646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:46.133337 11639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:46.134502 11639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:46.135687 11639 hybrid_clock.cc:648] HybridClock initialized: now 1790311006135667 us; error 31 us; skew 500 ppm
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:46.144850 11639 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:36:46.145771 11639 webserver.cc:460] Webserver started at http://127.27.46.254:36103/ using document root <none> and password file <none>
I20260925 04:36:46.146298 11639 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:46.146363 11639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:46.150068 11639 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:46.152071 11655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.153214 11639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:46.153451 11639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "1d81920204db474690740df5de2544d0"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "149409c7f6e2514462f3b6ee5e69ecd1"
server_key_iv: "6a088b8a5c64cc8f1599bcaeaa3be148"
server_key_version: "encryptionkey@0"
I20260925 04:36:46.154124 11639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:46.178296 11639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:46.180936 11639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:46.181130 11639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:46.187850 11639 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38215
I20260925 04:36:46.188144 11708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38215 every 8 connection(s)
I20260925 04:36:46.188863 11639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:36:46.192845 11709 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:36:46.194470 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap starting.
I20260925 04:36:46.196915 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11639
I20260925 04:36:46.197350 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:35007
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=40381
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:46.218104 11709 log.cc:824] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:46.229405 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:46.229871 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap complete.
I20260925 04:36:46.235606 11709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:46.236130 11709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d81920204db474690740df5de2544d0, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.236703 11709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:46.236850 11709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:36:46.236966 11709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:36:46.237066 11709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:36:46.239799 11709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:46.240208 11709 leader_election.cc:302] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d81920204db474690740df5de2544d0; no voters:
I20260925 04:36:46.240587 11709 leader_election.cc:288] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:36:46.240780 11713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:36:46.241032 11713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 LEADER]: Becoming Leader. State: Replica: 1d81920204db474690740df5de2544d0, State: Running, Role: LEADER
I20260925 04:36:46.241544 11713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:36:46.241995 11709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:36:46.243363 11715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:36:46.243533 11715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:46.243422 11714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d81920204db474690740df5de2544d0. Latest consensus state: current_term: 2 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:36:46.244457 11714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:36:46.244756 11724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:36:46.250320 11724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:36:46.250918 11724 catalog_manager.cc:1259] Loaded cluster ID: 3fe1ed79987a44e58a295492c86f6ce4
I20260925 04:36:46.250982 11724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:36:46.253284 11724 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:36:46.253954 11724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Loaded TSK: 0
I20260925 04:36:46.278424 11724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Generated new TSK 1
I20260925 04:36:46.278728 11724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:36:46.350442 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:46.350766 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:46.350840 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:46.356526 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:46.356606 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:46.356722 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:36:46.364101 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:35007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=40381
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:46.365376 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:46.366700 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:46.378616 11737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.378760 11739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.378625 11736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:46.379429 11711 server_base.cc:1034] running on GCE node
I20260925 04:36:46.379962 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:46.380916 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:46.382123 11711 hybrid_clock.cc:648] HybridClock initialized: now 1790311006382102 us; error 43 us; skew 500 ppm
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:46.385968 11711 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:36:46.387012 11711 webserver.cc:460] Webserver started at http://127.27.46.193:40381/ using document root <none> and password file <none>
I20260925 04:36:46.387691 11711 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:46.387797 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:46.392714 11711 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:46.395529 11747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.397082 11711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:46.397229 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "2209b5979dc24803ada11e45a668a61b"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "86ffcbc5ce0e398c0801c1f8bf9eafb3"
server_key_iv: "6a9c3e3b53aab6c50de07cfcc1aea810"
server_key_version: "encryptionkey@0"
I20260925 04:36:46.397965 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:46.430797 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:46.435300 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:46.435513 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:46.436285 11711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:46.437752 11754 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:36:46.443159 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:36:46.443235 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260925 04:36:46.443307 11711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:36:46.445575 11711 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:36:46.445659 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260925 04:36:46.446074 11754 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Bootstrap starting.
I20260925 04:36:46.457274 11711 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:35007
I20260925 04:36:46.458943 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:36:46.459683 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:35007 every 8 connection(s)
I20260925 04:36:46.463635 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11711
I20260925 04:36:46.464217 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:38071
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=41463
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:46.468241 11754 log.cc:824] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:46.481421 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.461978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:44699)
0925 04:36:46.464029 (+ 2051us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.464110 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.465479 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.467399 (+ 1920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.467407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.467895 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:46.468506 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.468524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.470458 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.470464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.471790 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.471800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.472924 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.472927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.473140 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.473870 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:46.473891 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:46.475812 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:46.478267 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.478273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.478285 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.478595 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.478971 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.478974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.478976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.479124 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.479791 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:46.479797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:46.480033 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.480194 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.480473 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":188,"threads_started":1}
I20260925 04:36:46.482676 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.461615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:44699 (local address 127.27.46.254:38215)
0925 04:36:46.463553 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.463574 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.464262 (+ 688us) server_negotiation.cc:407] Connection header received
0925 04:36:46.466055 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.466074 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.466582 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.466921 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.468706 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.470288 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.471933 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.472567 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.472714 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.476005 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.476011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.476025 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.476053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.478131 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.478708 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.478714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.478719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.478855 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.479219 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.479222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.479224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.479620 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.479729 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.481770 (+ 2041us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.482075 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":103,"threads_started":1}
I20260925 04:36:46.483186 11754 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:46.483680 11754 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Bootstrap complete.
I20260925 04:36:46.485118 11754 ts_tablet_manager.cc:1397] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.011s
I20260925 04:36:46.485316 11863 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:46.485572 11863 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:46.486074 11863 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
I20260925 04:36:46.488032 11754 raft_consensus.cc:348] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.488545 11754 raft_consensus.cc:729] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2209b5979dc24803ada11e45a668a61b, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.488742 11672 ts_manager.cc:194] Registered new tserver with Master: 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:46.489070 11754 consensus_queue.cc:260] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.490103 11754 ts_tablet_manager.cc:1428] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260925 04:36:46.494329 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:44699
I20260925 04:36:46.497186 11863 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
I20260925 04:36:46.501381 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.492009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.0.0.1:39812)
0925 04:36:46.492668 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.492713 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.492855 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.496453 (+ 3598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.496459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.496483 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:46.496600 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.496607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.498273 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.498278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.499643 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.499656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.500834 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.500838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.500963 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.501001 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.501118 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.501241 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":114,"threads_started":1}
I20260925 04:36:46.501381 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.492725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.27.46.193:35007)
0925 04:36:46.495617 (+ 2892us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.495632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.495667 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:36:46.495746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.495752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.496259 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.496438 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:46.497086 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.498133 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.499754 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.500530 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.500634 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.500769 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.501161 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.501249 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2748,"thread_start_us":89,"threads_started":1}
I20260925 04:36:46.502523 11795 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d21325a437442c91f5ffbd2238209f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:39812
I20260925 04:36:46.503612 11872 tablet_replica.cc:330] stopping tablet replica
I20260925 04:36:46.503937 11872 raft_consensus.cc:2229] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:46.504187 11872 raft_consensus.cc:2258] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:46.505363 11872 ts_tablet_manager.cc:1905] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:46.510059 11872 ts_tablet_manager.cc:1918] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:46.510157 11872 log.cc:1196] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal/wals/d9d21325a437442c91f5ffbd2238209f
I20260925 04:36:46.510450 11872 ts_tablet_manager.cc:1939] T d9d21325a437442c91f5ffbd2238209f P 2209b5979dc24803ada11e45a668a61b: Deleting consensus metadata
I20260925 04:36:46.512091 11656 catalog_manager.cc:4841] TS 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007): tablet d9d21325a437442c91f5ffbd2238209f (table test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]) successfully deleted
W20260925 04:36:46.592773 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:46.592990 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:46.593029 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:46.597002 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:46.597054 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:46.597149 11867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:36:46.602015 11867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:38071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=41463
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:46.603009 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:46.604040 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:46.613272 11877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.613292 11880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.613308 11878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:46.613474 11867 server_base.cc:1034] running on GCE node
I20260925 04:36:46.613914 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:46.614550 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:46.615721 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790311006615690 us; error 44 us; skew 500 ppm
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:46.618324 11867 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:36:46.619210 11867 webserver.cc:460] Webserver started at http://127.27.46.194:41463/ using document root <none> and password file <none>
I20260925 04:36:46.619762 11867 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:46.619884 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:46.623533 11867 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:46.625761 11887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.626910 11867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:46.627058 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "f446dda56bee8bd6592342186ff0a78a"
server_key_iv: "a4f13f7341173bb3bf045f5727776e42"
server_key_version: "encryptionkey@0"
I20260925 04:36:46.627553 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:46.647374 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:46.649953 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:46.650151 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:46.650687 11867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:46.651738 11894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:36:46.655710 11867 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:36:46.655774 11867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 04:36:46.655844 11867 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:36:46.657554 11867 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:36:46.657624 11867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:46.657858 11894 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Bootstrap starting.
I20260925 04:36:46.664521 11894 log.cc:824] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:46.667838 11867 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:38071
I20260925 04:36:46.667995 12003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:38071 every 8 connection(s)
I20260925 04:36:46.668691 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:36:46.672505 11894 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:36:46.672911 11894 tablet_bootstrap.cc:492] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Bootstrap complete.
I20260925 04:36:46.673290 11894 ts_tablet_manager.cc:1397] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260925 04:36:46.676112 11894 raft_consensus.cc:348] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.676543 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 11867
I20260925 04:36:46.676631 11894 raft_consensus.cc:729] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92bd3ecf5e24096bdf409fb6cd794af, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.677016 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:39221
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=33457
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:36:46.677126 11894 consensus_queue.cc:260] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.678535 11894 ts_tablet_manager.cc:1428] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Time spent starting tablet: real 0.005s user 0.002s sys 0.006s
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:46.683137 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.670483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:44211 (local address 127.27.46.254:38215)
0925 04:36:46.670639 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.670643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.671280 (+ 637us) server_negotiation.cc:407] Connection header received
0925 04:36:46.672348 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.672351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.672400 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.672474 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.673973 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.674508 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.676060 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.676295 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.676413 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.679618 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.679624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.679627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.679660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.681073 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.681595 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.681598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.681600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.681715 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.682062 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.682065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.682067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.682260 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.682365 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.682831 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.682962 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:36:46.683948 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.670616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:44211)
0925 04:36:46.671112 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.671165 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.672139 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.672643 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.672653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.673146 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:46.673786 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.673823 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.674654 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.674657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.675940 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.675950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.676429 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.676433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.676643 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.677718 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:46.677734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:46.679436 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:46.681206 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.681211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.681225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.681455 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.681819 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.681823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.681825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.681958 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.682363 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:46.682370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:46.682592 (+ 222us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.682772 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.683072 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":118,"threads_started":1}
I20260925 04:36:46.685250 12004 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:46.685467 12004 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:46.685949 12004 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
I20260925 04:36:46.687533 11672 ts_manager.cc:194] Registered new tserver with Master: d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
I20260925 04:36:46.688609 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:44211
I20260925 04:36:46.690765 12004 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.194@KRBTEST.COM
I20260925 04:36:46.697849 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.688339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38071 (local address 127.0.0.1:48946)
0925 04:36:46.688459 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.688472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.688548 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.689181 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.689184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.689608 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:46.689720 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.689727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.690931 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.690935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.691725 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.691735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.692422 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.692424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.692486 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.692939 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:46.692949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:46.694266 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:46.696247 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.696256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.696260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.696482 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.696796 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.696798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.696799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.696880 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.697363 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:46.697367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:46.697465 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.697637 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.697745 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 04:36:46.698032 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.688474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.27.46.194:38071)
0925 04:36:46.688751 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.688762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.688783 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:36:46.688840 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.688844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.689048 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.689344 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.689839 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.690761 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.691849 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.692320 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.692409 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.694401 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.694411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.694424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.694448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.696135 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.696594 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.696603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.696606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.696696 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.696976 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.696979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.696980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.697255 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.697372 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.697727 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.697875 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260925 04:36:46.698863 11936 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d21325a437442c91f5ffbd2238209f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:48946
I20260925 04:36:46.699896 12010 tablet_replica.cc:330] stopping tablet replica
I20260925 04:36:46.700178 12010 raft_consensus.cc:2229] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:46.700415 12010 raft_consensus.cc:2258] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:46.701309 12010 ts_tablet_manager.cc:1905] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:46.704946 12010 ts_tablet_manager.cc:1918] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:46.705034 12010 log.cc:1196] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/d9d21325a437442c91f5ffbd2238209f
I20260925 04:36:46.705282 12010 ts_tablet_manager.cc:1939] T d9d21325a437442c91f5ffbd2238209f P d92bd3ecf5e24096bdf409fb6cd794af: Deleting consensus metadata
I20260925 04:36:46.706892 11657 catalog_manager.cc:4841] TS d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): tablet d9d21325a437442c91f5ffbd2238209f (table test-table [id=a3afbb39cb3c4a8bb83edb1edfa42d52]) successfully deleted
W20260925 04:36:46.792574 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:36:46.792804 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:36:46.792843 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:36:46.796831 12008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:36:46.796890 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:36:46.796962 12008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:36:46.802109 12008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:39221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=33457
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:36:46.803112 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:36:46.804199 12008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:36:46.813736 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.813808 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:36:46.813779 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:36:46.813791 12008 server_base.cc:1034] running on GCE node
I20260925 04:36:46.814356 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:36:46.814960 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:36:46.816118 12008 hybrid_clock.cc:648] HybridClock initialized: now 1790311006816091 us; error 38 us; skew 500 ppm
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:36:46.818646 12008 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:36:46.819499 12008 webserver.cc:460] Webserver started at http://127.27.46.195:33457/ using document root <none> and password file <none>
I20260925 04:36:46.820034 12008 fs_manager.cc:359] Metadata directory not provided
I20260925 04:36:46.820124 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:36:46.823781 12008 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:36:46.825871 12025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.827054 12008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:36:46.827188 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "83947d33083d403395a91ab4db50945d"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "0e3e4c6edf7b332f77b9473b4034ada0"
server_key_iv: "4e533259a4c98a47398a2bda360a13ed"
server_key_version: "encryptionkey@0"
I20260925 04:36:46.827708 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:36:46.839617 12008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:36:46.842521 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:36:46.842788 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:36:46.843456 12008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:36:46.844290 12008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:36:46.844355 12008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.844419 12008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:36:46.844466 12008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:36:46.853174 12008 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:39221
I20260925 04:36:46.853201 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:39221 every 8 connection(s)
I20260925 04:36:46.854176 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:36:46.854604 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12008
I20260925 04:36:46.864121 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.855312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.27.46.254:38215)
0925 04:36:46.855445 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.855452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.855689 (+ 237us) server_negotiation.cc:407] Connection header received
0925 04:36:46.855889 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.855893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.855952 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.856042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.856774 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.857299 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.858541 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.858836 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.858954 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.860664 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.860670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.860673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.860700 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.862338 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.862761 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.862764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.862766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.862852 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.863126 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.863128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.863130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.863359 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.863563 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.863792 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.863919 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 04:36:46 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:36:46.868680 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.855977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:52737 (local address 127.27.46.254:38215)
0925 04:36:46.857367 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.857371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.857384 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:36:46.858275 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.858279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.858332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.860146 (+ 1814us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.860283 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.861029 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.862278 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.862580 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.862651 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.865287 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.865290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.865293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.865316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.866759 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.867209 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.867212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.867213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.867308 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.867631 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.867635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.867636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.867797 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.867916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.868402 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.868511 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":59,"threads_started":1}
I20260925 04:36:46.869410 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.856391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:52737)
0925 04:36:46.857076 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.857121 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.858050 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.858742 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.858753 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.859269 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:36:46.859977 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.860001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.861183 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.861186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.862150 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.862160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.862730 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.862734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.862942 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.863506 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:36:46.863526 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:36:46.865099 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:36:46.866864 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.866869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.866882 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.867107 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.867410 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:36:46.867413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:36:46.867415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:36:46.867538 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:36:46.867915 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:36:46.867922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:36:46.868173 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.868358 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.868655 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":128,"threads_started":1}
I20260925 04:36:46.870486 12139 heartbeater.cc:347] Connected to a master server at 127.27.46.254:38215
I20260925 04:36:46.870712 12139 heartbeater.cc:464] Registering TS with master...
I20260925 04:36:46.871228 12139 heartbeater.cc:511] Master 127.27.46.254:38215 requested a full tablet report, sending...
I20260925 04:36:46.872303 11672 ts_manager.cc:194] Registered new tserver with Master: 83947d33083d403395a91ab4db50945d (127.27.46.195:39221)
I20260925 04:36:46.872879 11672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:52737
I20260925 04:36:46.878185 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:36:46.881840 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.878766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.27.46.254:38215)
0925 04:36:46.878894 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.878898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.879084 (+ 186us) server_negotiation.cc:407] Connection header received
0925 04:36:46.879210 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.879212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.879298 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.879374 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:46.879600 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.880058 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.880807 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.881009 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.881081 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.881333 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:46.881442 (+ 109us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 04:36:46.881708 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47018: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 04:36:46.881906 12142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47018: token expired
W20260925 04:36:46.882216 11599 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.46.254:38215)
I20260925 04:36:46.888460 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.882530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.27.46.254:38215)
0925 04:36:46.882634 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.882638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.882655 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:36:46.882740 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.882742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.882770 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.882824 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:36:46.883294 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.883774 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.884480 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.884667 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.884746 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.886079 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:36:46.886085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:36:46.886089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:36:46.886119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:36:46.887319 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.887649 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.887651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.887652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.887715 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:36:46.887928 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:36:46.887930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:36:46.887931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:36:46.888060 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:36:46.888147 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.888290 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.888374 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:36:46.890573 11672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:36:46.893760 11672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:36:46.901024 11795 tablet_service.cc:1467] Processing CreateTablet for tablet 356c661f67254fb1b66cfee6d980dada (DEFAULT_TABLE table=test-table [id=43087604cae34c459bee29c894649104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:46.901069 11936 tablet_service.cc:1467] Processing CreateTablet for tablet 356c661f67254fb1b66cfee6d980dada (DEFAULT_TABLE table=test-table [id=43087604cae34c459bee29c894649104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:46.901655 11936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c661f67254fb1b66cfee6d980dada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:46.901671 11795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c661f67254fb1b66cfee6d980dada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:46.904500 11894 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Bootstrap starting.
I20260925 04:36:46.904557 11754 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Bootstrap starting.
I20260925 04:36:46.905400 11754 tablet_bootstrap.cc:654] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:46.905411 11894 tablet_bootstrap.cc:654] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:46.905788 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.900162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39221 (local address 127.0.0.1:59806)
0925 04:36:46.900283 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:46.900299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:46.900378 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:46.901294 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:46.901296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:46.901313 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:46.901413 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.901419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.903072 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.903076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.904304 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:46.904316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.905350 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.905353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:46.905446 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.905467 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:36:46.905579 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:36:46.905660 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 04:36:46.906293 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:46.900384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.27.46.195:39221)
0925 04:36:46.900812 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:46.900824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:46.900847 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:36:46.900921 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:46.900926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:46.901131 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:46.901596 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:46.901776 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.902949 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.904440 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:46.905216 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:46.905340 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:46.905486 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:46.906005 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 04:36:46.906123 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260925 04:36:46.906706 11754 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: No bootstrap required, opened a new log
I20260925 04:36:46.906796 11754 ts_tablet_manager.cc:1397] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 04:36:46.907356 11894 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: No bootstrap required, opened a new log
I20260925 04:36:46.907440 11894 ts_tablet_manager.cc:1397] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:36:46.907521 11754 raft_consensus.cc:348] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.907619 11754 raft_consensus.cc:374] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:46.907644 11754 raft_consensus.cc:729] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2209b5979dc24803ada11e45a668a61b, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.907785 11754 consensus_queue.cc:260] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.907872 11894 raft_consensus.cc:348] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.907972 11894 raft_consensus.cc:374] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:46.908007 11894 raft_consensus.cc:729] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92bd3ecf5e24096bdf409fb6cd794af, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.908062 11754 ts_tablet_manager.cc:1428] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 04:36:46.908118 11894 consensus_queue.cc:260] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.908355 11894 ts_tablet_manager.cc:1428] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 04:36:46.909729 12073 tablet_service.cc:1467] Processing CreateTablet for tablet 356c661f67254fb1b66cfee6d980dada (DEFAULT_TABLE table=test-table [id=43087604cae34c459bee29c894649104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:36:46.910984 12073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356c661f67254fb1b66cfee6d980dada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:36:46.917339 12151 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Bootstrap starting.
W20260925 04:36:46.919572 12005 tablet.cc:2369] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:46.919726 12151 tablet_bootstrap.cc:654] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Neither blocks nor log segments found. Creating new log.
I20260925 04:36:46.920409 12151 log.cc:824] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Log is configured to *not* fsync() on all Append() calls
I20260925 04:36:46.922060 12151 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: No bootstrap required, opened a new log
I20260925 04:36:46.922250 12151 ts_tablet_manager.cc:1397] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:36:46.925187 12151 raft_consensus.cc:348] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.925415 12151 raft_consensus.cc:374] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:36:46.925469 12151 raft_consensus.cc:729] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83947d33083d403395a91ab4db50945d, State: Initialized, Role: FOLLOWER
I20260925 04:36:46.925906 12151 consensus_queue.cc:260] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:46.926491 12139 heartbeater.cc:503] Master 127.27.46.254:38215 was elected leader, sending a full tablet report...
I20260925 04:36:46.926790 12151 ts_tablet_manager.cc:1428] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260925 04:36:46.968151 11864 tablet.cc:2369] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:36:47.105307 12140 tablet.cc:2369] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:36:47.139518 12007 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:36:47.139704 12007 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:47.140935 12007 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:47.145463 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.141215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:34841 (local address 127.27.46.195:39221)
0925 04:36:47.141359 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:47.141363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:47.141379 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:36:47.141528 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:47.141532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:47.141587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:47.141673 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:47.141967 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.142829 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.144315 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.144884 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.145003 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:47.145057 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:47.145242 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 04:36:47.145322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:36:47.145449 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.141115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39221 (local address 127.27.46.194:34841)
0925 04:36:47.141239 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:47.141256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:47.141360 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:47.141701 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:47.141704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:47.141724 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:47.141845 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:47.141849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.142930 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.142934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:47.144195 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:47.144205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.144998 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.145000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:47.145080 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:47.145117 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:36:47.145211 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:36:47.145295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 04:36:47.145725 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.141492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:52629 (local address 127.27.46.193:35007)
0925 04:36:47.141960 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:47.141965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:47.141978 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:36:47.142030 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:47.142032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:47.142078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:47.142183 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:36:47.142487 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.143137 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.144498 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.145109 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.145230 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:47.145288 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:47.145453 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:36:47.145585 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":135,"threads_started":1}
I20260925 04:36:47.145828 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.141192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.27.46.194:52629)
0925 04:36:47.141449 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:47.141462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:36:47.141625 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:36:47.142200 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:36:47.142202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:36:47.142218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:36:47.142377 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:47.142383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.143239 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.143243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:47.144394 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:36:47.144401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.145241 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.145244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:36:47.145332 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:47.145353 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:36:47.145422 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:36:47.145522 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260925 04:36:47.146335 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c661f67254fb1b66cfee6d980dada" candidate_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83947d33083d403395a91ab4db50945d" is_pre_election: true
I20260925 04:36:47.146402 11803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c661f67254fb1b66cfee6d980dada" candidate_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2209b5979dc24803ada11e45a668a61b" is_pre_election: true
I20260925 04:36:47.146677 11803 raft_consensus.cc:2454] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92bd3ecf5e24096bdf409fb6cd794af in term 0.
I20260925 04:36:47.146677 12093 raft_consensus.cc:2454] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92bd3ecf5e24096bdf409fb6cd794af in term 0.
I20260925 04:36:47.147217 11888 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b, 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af; no voters:
I20260925 04:36:47.147507 12007 raft_consensus.cc:2781] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:36:47.147603 12007 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:36:47.147677 12007 raft_consensus.cc:3037] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:47.148695 12007 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:47.149049 12007 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [CANDIDATE]: Term 1 election: Requested vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007)
I20260925 04:36:47.149415 12093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c661f67254fb1b66cfee6d980dada" candidate_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83947d33083d403395a91ab4db50945d"
I20260925 04:36:47.149415 11803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356c661f67254fb1b66cfee6d980dada" candidate_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2209b5979dc24803ada11e45a668a61b"
I20260925 04:36:47.149564 12093 raft_consensus.cc:3037] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:47.149565 11803 raft_consensus.cc:3037] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:36:47.150738 12093 raft_consensus.cc:2454] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92bd3ecf5e24096bdf409fb6cd794af in term 1.
I20260925 04:36:47.150745 11803 raft_consensus.cc:2454] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92bd3ecf5e24096bdf409fb6cd794af in term 1.
I20260925 04:36:47.151046 11890 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b, 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af; no voters:
I20260925 04:36:47.151243 12007 raft_consensus.cc:2781] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:36:47.151415 12007 raft_consensus.cc:686] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 1 LEADER]: Becoming Leader. State: Replica: d92bd3ecf5e24096bdf409fb6cd794af, State: Running, Role: LEADER
I20260925 04:36:47.151655 12007 consensus_queue.cc:237] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:36:47.154484 11671 catalog_manager.cc:5495] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af reported cstate change: term changed from 0 to 1, leader changed from <none> to d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194). New cstate: current_term: 1 leader_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } health_report { overall_health: UNKNOWN } } }
I20260925 04:36:47.200011 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.196039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.27.46.194:38071)
0925 04:36:47.196180 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:36:47.196185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:36:47.196296 (+ 111us) server_negotiation.cc:407] Connection header received
0925 04:36:47.196423 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:36:47.196426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:36:47.196490 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:36:47.196600 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:36:47.196831 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.197732 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.198495 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:36:47.198837 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:36:47.198942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:36:47.199162 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:36:47.199649 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:36:47.199758 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:36:47.199820 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 04:36:47.199886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:36:47.204316 12093 raft_consensus.cc:1261] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Refusing update from remote peer d92bd3ecf5e24096bdf409fb6cd794af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:47.204335 11803 raft_consensus.cc:1261] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Refusing update from remote peer d92bd3ecf5e24096bdf409fb6cd794af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:36:47.205001 12007 consensus_queue.cc:1035] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [LEADER]: Connected to new peer: Peer: permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:47.205183 12157 consensus_queue.cc:1035] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [LEADER]: Connected to new peer: Peer: permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:36:47.210515 12161 mvcc.cc:205] Tried to move back new op lower bound from 7333113885502345216 to 7333113885295607808. Current Snapshot: MvccSnapshot[applied={T|T < 7333113885502345216}]
I20260925 04:36:47.215132 12162 mvcc.cc:205] Tried to move back new op lower bound from 7333113885502345216 to 7333113885295607808. Current Snapshot: MvccSnapshot[applied={T|T < 7333113885502345216}]
I20260925 04:36:47.219806 11671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:47.219942 11671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:36:47.222214 11671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Sending DeleteTablet for 3 replicas of tablet 356c661f67254fb1b66cfee6d980dada
I20260925 04:36:47.222860 12073 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c661f67254fb1b66cfee6d980dada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:47 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:59806
I20260925 04:36:47.222946 11936 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c661f67254fb1b66cfee6d980dada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:47 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:48946
I20260925 04:36:47.222957 11795 tablet_service.cc:1514] Processing DeleteTablet for tablet 356c661f67254fb1b66cfee6d980dada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:36:47 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:39812
I20260925 04:36:47.223259 12171 tablet_replica.cc:330] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: stopping tablet replica
I20260925 04:36:47.223398 12171 raft_consensus.cc:2229] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:47.223564 12171 raft_consensus.cc:2258] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:47.223644 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11711
I20260925 04:36:47.223845 12172 tablet_replica.cc:330] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: stopping tablet replica
I20260925 04:36:47.224118 12172 raft_consensus.cc:2229] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:36:47.224443 12172 raft_consensus.cc:2258] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:47.225922 12172 ts_tablet_manager.cc:1905] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:47.228256 12172 ts_tablet_manager.cc:1918] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:47.228364 12172 log.cc:1196] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/356c661f67254fb1b66cfee6d980dada
I20260925 04:36:47.228722 12172 ts_tablet_manager.cc:1939] T 356c661f67254fb1b66cfee6d980dada P 83947d33083d403395a91ab4db50945d: Deleting consensus metadata
I20260925 04:36:47.229557 12170 tablet_replica.cc:330] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: stopping tablet replica
I20260925 04:36:47.229732 12170 raft_consensus.cc:2229] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:36:47.230000 11658 catalog_manager.cc:4841] TS 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): tablet 356c661f67254fb1b66cfee6d980dada (table test-table [id=43087604cae34c459bee29c894649104]) successfully deleted
I20260925 04:36:47.230163 12170 raft_consensus.cc:2258] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:36:47.231115 12170 ts_tablet_manager.cc:1905] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:36:47.233297 12170 ts_tablet_manager.cc:1918] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:36:47.233389 12170 log.cc:1196] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/356c661f67254fb1b66cfee6d980dada
I20260925 04:36:47.233732 12170 ts_tablet_manager.cc:1939] T 356c661f67254fb1b66cfee6d980dada P d92bd3ecf5e24096bdf409fb6cd794af: Deleting consensus metadata
I20260925 04:36:47.234361 11657 catalog_manager.cc:4841] TS d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): tablet 356c661f67254fb1b66cfee6d980dada (table test-table [id=43087604cae34c459bee29c894649104]) successfully deleted
W20260925 04:36:47.236016 11656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:35007 (error 108)
I20260925 04:36:47.236250 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11867
I20260925 04:36:47.237412 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:36:47.237127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:35007 (local address 127.0.0.1:39816)
0925 04:36:47.237258 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:36:47.237343 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:35007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:36:47.237660 11656 catalog_manager.cc:4584] TS 2209b5979dc24803ada11e45a668a61b (127.27.46.193:35007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 356c661f67254fb1b66cfee6d980dada: Network error: Client connection negotiation failed: client connection to 127.27.46.193:35007: connect: Connection refused (error 111)
I20260925 04:36:47.242806 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12008
I20260925 04:36:47.248741 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 11639
I20260925 04:37:07.255385 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--webserver_interface=127.27.46.254
--webserver_port=36103
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:07.363852 12174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:07.364089 12174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:07.364127 12174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:07.368000 12174 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:37:07.368041 12174 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:37:07.368055 12174 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:37:07.368067 12174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:07.368083 12174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:07.368098 12174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:07.368112 12174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:07.368125 12174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:07.372817 12174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=36103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:07.373723 12174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:07.374666 12174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:07.383011 12179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:07.383019 12182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:07.383019 12180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.383181 12174 server_base.cc:1034] running on GCE node
I20260925 04:37:07.383893 12174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:07.384945 12174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:07.386142 12174 hybrid_clock.cc:648] HybridClock initialized: now 1790311027386128 us; error 35 us; skew 500 ppm
Sep 25 04:37:07 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:07.388645 12174 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:07.389564 12174 webserver.cc:460] Webserver started at http://127.27.46.254:36103/ using document root <none> and password file <none>
I20260925 04:37:07.390089 12174 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:07.390182 12174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:07.393937 12174 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 04:37:07.396018 12189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.397209 12174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.397346 12174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "1d81920204db474690740df5de2544d0"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "149409c7f6e2514462f3b6ee5e69ecd1"
server_key_iv: "6a088b8a5c64cc8f1599bcaeaa3be148"
server_key_version: "encryptionkey@0"
I20260925 04:37:07.397959 12174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:07.418780 12174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:07.421867 12174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:07.422075 12174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:07.429708 12174 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38215
I20260925 04:37:07.430035 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38215 every 8 connection(s)
I20260925 04:37:07.430905 12174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:07.431751 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12174
I20260925 04:37:07.432178 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:35007
--local_ip_for_outbound_sockets=127.27.46.193
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=40381
--webserver_interface=127.27.46.193
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:37:07.435310 12245 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:37:07.436911 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap starting.
I20260925 04:37:07.454128 12245 log.cc:824] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:07.464596 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:37:07.464967 12245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Bootstrap complete.
I20260925 04:37:07.468780 12245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:37:07.469159 12245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d81920204db474690740df5de2544d0, State: Initialized, Role: FOLLOWER
I20260925 04:37:07.469674 12245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:37:07.469821 12245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:07.469901 12245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:07.470009 12245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:37:07.471766 12245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:37:07.472086 12245 leader_election.cc:302] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d81920204db474690740df5de2544d0; no voters:
I20260925 04:37:07.472401 12245 leader_election.cc:288] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:37:07.472532 12249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:37:07.472844 12249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [term 3 LEADER]: Becoming Leader. State: Replica: 1d81920204db474690740df5de2544d0, State: Running, Role: LEADER
I20260925 04:37:07.473263 12249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } }
I20260925 04:37:07.473582 12245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:07.474768 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:37:07.474875 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:07.475596 12259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:07.475548 12251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d81920204db474690740df5de2544d0. Latest consensus state: current_term: 3 leader_uuid: "1d81920204db474690740df5de2544d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d81920204db474690740df5de2544d0" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38215 } } }
I20260925 04:37:07.475759 12251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:07.480487 12259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:07.481106 12259 catalog_manager.cc:1259] Loaded cluster ID: 3fe1ed79987a44e58a295492c86f6ce4
I20260925 04:37:07.481163 12259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:07.483686 12259 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:07.484423 12259 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Loaded TSK: 0
I20260925 04:37:07.484509 12259 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d81920204db474690740df5de2544d0: Loaded TSK: 1
F20260925 04:37:07.485265 12259 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0c9cff49ad google::LogMessage::Fail() at ??:0
@ 0x7f0c9cff788a google::LogMessage::SendToLog() at ??:0
@ 0x7f0c9cff44de google::LogMessage::Flush() at ??:0
@ 0x7f0c9cff5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c9fcb818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f0c9fcb0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f0ca79597bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f0ca793399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f0ca7934178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f0ca7975df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f0ca799482b std::function<>::operator()() at ??:0
@ 0x7f0ca7933b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0ca7934f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0ca7931579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f0ca7975640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0c9f2463d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0c9f246c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0c9f2483f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f0c9f237512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0ca087b6db start_thread at ??:0
@ 0x7f0c9baf261f clone at ??:0
W20260925 04:37:07.578387 12247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:07.578596 12247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:07.578658 12247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:07.582875 12247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:07.582921 12247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:07.582991 12247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:07.587949 12247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:35007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=40381
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:07.588860 12247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:07.589798 12247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:07.598469 12292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:07.598469 12293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.598600 12247 server_base.cc:1034] running on GCE node
W20260925 04:37:07.598706 12295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.599100 12247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:07.599750 12247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:07.600917 12247 hybrid_clock.cc:648] HybridClock initialized: now 1790311027600892 us; error 36 us; skew 500 ppm
Sep 25 04:37:07 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:07.603395 12247 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:37:07.604302 12247 webserver.cc:460] Webserver started at http://127.27.46.193:40381/ using document root <none> and password file <none>
I20260925 04:37:07.604828 12247 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:07.604902 12247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:07.608600 12247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.612321 12302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.613651 12247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.613770 12247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "2209b5979dc24803ada11e45a668a61b"
format_stamp: "Formatted at 2026-09-25 04:36:24 on dist-test-slave-r8r3"
server_key: "86ffcbc5ce0e398c0801c1f8bf9eafb3"
server_key_iv: "6a9c3e3b53aab6c50de07cfcc1aea810"
server_key_version: "encryptionkey@0"
I20260925 04:37:07.614281 12247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:07.628131 12247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:07.630920 12247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:07.631125 12247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:07.631811 12247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:07.632899 12310 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 04:37:07.636893 12247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:37:07.636950 12247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 04:37:07.636993 12247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:37:07.638686 12247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:37:07.638756 12247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 04:37:07.639127 12310 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Bootstrap starting.
I20260925 04:37:07.649413 12247 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:35007
I20260925 04:37:07.650087 12419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:35007 every 8 connection(s)
I20260925 04:37:07.650956 12247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:07.659600 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12247
I20260925 04:37:07.660081 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:38071
--local_ip_for_outbound_sockets=127.27.46.194
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=41463
--webserver_interface=127.27.46.194
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:37:07.660558 12310 log.cc:824] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:07.669339 12310 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:37:07.669778 12310 tablet_bootstrap.cc:492] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Bootstrap complete.
I20260925 04:37:07.671257 12310 ts_tablet_manager.cc:1397] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.007s
I20260925 04:37:07.674387 12310 raft_consensus.cc:348] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:07.674909 12310 raft_consensus.cc:729] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2209b5979dc24803ada11e45a668a61b, State: Initialized, Role: FOLLOWER
I20260925 04:37:07.675395 12310 consensus_queue.cc:260] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:07.676502 12310 ts_tablet_manager.cc:1428] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
W20260925 04:37:07.707119 12422 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:37:07.652706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:51951)
0925 04:37:07.653126 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.653172 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:07.653925 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:07.706372 (+ 52447us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
W20260925 04:37:07.707620 12420 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38215 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:38215: Client connection negotiation failed: client connection to 127.27.46.254:38215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:37:07.708400 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:07.708158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:53813)
0925 04:37:07.708284 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.708345 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:37:07.709092 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:07.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:54297)
0925 04:37:07.708969 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.709028 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:37:07.709355 12420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:37:07.784879 12424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:07.785176 12424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:07.785267 12424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:07.789520 12424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:07.789793 12424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:07.789923 12424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:07.795729 12424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:38071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=41463
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:07.796798 12424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:07.797861 12424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:07.807291 12430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:07.807361 12433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.807351 12424 server_base.cc:1034] running on GCE node
W20260925 04:37:07.807304 12431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.807907 12424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:07.808512 12424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:07.809650 12424 hybrid_clock.cc:648] HybridClock initialized: now 1790311027809620 us; error 42 us; skew 500 ppm
Sep 25 04:37:07 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:07.812181 12424 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:37:07.813035 12424 webserver.cc:460] Webserver started at http://127.27.46.194:41463/ using document root <none> and password file <none>
I20260925 04:37:07.813634 12424 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:07.813697 12424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:07.817332 12424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:07.819393 12440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.820572 12424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.820691 12424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "d92bd3ecf5e24096bdf409fb6cd794af"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "f446dda56bee8bd6592342186ff0a78a"
server_key_iv: "a4f13f7341173bb3bf045f5727776e42"
server_key_version: "encryptionkey@0"
I20260925 04:37:07.821203 12424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:07.835479 12424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:07.838176 12424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:07.838367 12424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:07.838912 12424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:07.839639 12424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:07.839699 12424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.839762 12424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:07.839823 12424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.848608 12424 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:38071
I20260925 04:37:07.848645 12555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:38071 every 8 connection(s)
I20260925 04:37:07.849563 12424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:07.852780 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:07.851431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:42365)
0925 04:37:07.852023 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.852119 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":92,"threads_started":1}
W20260925 04:37:07.853277 12556 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38215 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:38215: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
I20260925 04:37:07.853757 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12424
I20260925 04:37:07.853919 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:07.853654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:54147)
0925 04:37:07.853781 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.853854 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:37:07.854277 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:39221
--local_ip_for_outbound_sockets=127.27.46.195
--tserver_master_addrs=127.27.46.254:38215
--webserver_port=33457
--webserver_interface=127.27.46.195
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:37:07.854640 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:07.854393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:44401)
0925 04:37:07.854548 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:07.854585 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260925 04:37:07.854879 12556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:37:07.965919 12559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:07.966149 12559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:07.966218 12559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:07.969897 12559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:07.969942 12559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:07.970031 12559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:07.974741 12559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:39221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=33457
--tserver_master_addrs=127.27.46.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:07.975649 12559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:07.976800 12559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:07.985906 12565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.985970 12559 server_base.cc:1034] running on GCE node
W20260925 04:37:07.985896 12564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:07.986052 12567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:07.986487 12559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:07.987109 12559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:07.988363 12559 hybrid_clock.cc:648] HybridClock initialized: now 1790311027988351 us; error 38 us; skew 500 ppm
Sep 25 04:37:07 dist-test-slave-r8r3 krb5kdc[11086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:07.991108 12559 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:37:07.991948 12559 webserver.cc:460] Webserver started at http://127.27.46.195:33457/ using document root <none> and password file <none>
I20260925 04:37:07.992462 12559 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:07.992520 12559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:07.996227 12559 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.998162 12574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:07.999312 12559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:07.999428 12559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "83947d33083d403395a91ab4db50945d"
format_stamp: "Formatted at 2026-09-25 04:36:25 on dist-test-slave-r8r3"
server_key: "0e3e4c6edf7b332f77b9473b4034ada0"
server_key_iv: "4e533259a4c98a47398a2bda360a13ed"
server_key_version: "encryptionkey@0"
I20260925 04:37:07.999934 12559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:08.041486 12559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:08.044301 12559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:08.044504 12559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:08.045138 12559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:08.045984 12559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:08.046030 12559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:08.046098 12559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:08.046144 12559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:08.054761 12559 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:39221
I20260925 04:37:08.054791 12688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:39221 every 8 connection(s)
I20260925 04:37:08.055759 12559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:08.059254 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.057810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:47605)
0925 04:37:08.058462 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.058564 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":108,"threads_started":1}
W20260925 04:37:08.059726 12689 heartbeater.cc:644] Failed to heartbeat to 127.27.46.254:38215 (0 consecutive failures): Network error: Failed to ping master at 127.27.46.254:38215: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
I20260925 04:37:08.060264 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.060077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:52953)
0925 04:37:08.060172 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.060214 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 04:37:08.060622 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12559
I20260925 04:37:08.060894 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.060709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:51193)
0925 04:37:08.060802 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.060845 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 04:37:08.061080 12689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:37:08.710354 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.709932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:42075)
0925 04:37:08.710159 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.710233 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260925 04:37:08.855731 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.855363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:37295)
0925 04:37:08.855587 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.855637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260925 04:37:08.967206 12695 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:08.967438 12695 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:08.968557 12695 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
Sep 25 04:37:08 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.195@KRBTEST.COM
Sep 25 04:37:08 dist-test-slave-r8r3 krb5kdc[11086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.194@KRBTEST.COM
I20260925 04:37:08.983443 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.968771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:38071 (local address 127.27.46.193:60823)
0925 04:37:08.968877 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.968893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:08.968999 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:08.971268 (+ 2269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:08.971273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:08.971814 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:08.972347 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:08.972364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.973705 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.973710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:08.974664 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:08.974675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.975404 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.975407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:08.975522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:08.976222 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:08.976239 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:08.978250 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:08.980833 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:08.980837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:08.980847 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:08.981202 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:08.981611 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:08.981616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:08.981740 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:08.982356 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:08.982363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:08.982650 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 04:37:08.982874 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 04:37:08.983214 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":198,"spinlock_wait_cycles":40704}
I20260925 04:37:08.983546 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.968967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60823 (local address 127.27.46.194:38071)
0925 04:37:08.969355 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:08.969376 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:08.969419 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:37:08.970366 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:08.970374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:08.970822 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:08.971163 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:08.972533 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.973576 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.974797 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.975270 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.975450 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:08.978437 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:08.978445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:08.978462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:08.978498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:08.980716 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981338 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:08.981344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:08.981354 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:08.981491 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981830 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:08.981834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:08.981836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:08.982235 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:08.982369 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:08.983012 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 04:37:08.983313 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":69,"threads_started":1}
I20260925 04:37:08.983561 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.968771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39221 (local address 127.27.46.193:60043)
0925 04:37:08.969406 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:08.969417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:08.969551 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:08.971646 (+ 2095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:08.971649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:08.971892 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:08.972346 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:08.972362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.973687 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.973695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:08.974659 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:08.974668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.975363 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.975366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:08.975519 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:08.976265 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:08.976286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:08.977837 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:08.980877 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:08.980880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:08.980882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:08.981193 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:08.981633 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:08.981638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:08.981751 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:08.982427 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:08.982429 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:08.982650 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 04:37:08.982874 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 04:37:08.983318 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":556,"spinlock_wait_cycles":21888,"thread_start_us":65,"threads_started":1}
I20260925 04:37:08.984413 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:08.969015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:60043 (local address 127.27.46.195:39221)
0925 04:37:08.969355 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:08.969373 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:08.969525 (+ 152us) server_negotiation.cc:407] Connection header received
0925 04:37:08.970365 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:08.970374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:08.970822 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:08.971163 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:08.972509 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.973485 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.974774 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:08.975218 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:08.975408 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:08.978438 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:08.978445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:08.978462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:08.978498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:08.980726 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981328 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:08.981335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:08.981340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:08.981495 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:08.981857 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:08.981861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:08.981863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:08.982280 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:08.982421 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:08.983285 (+ 864us) server_negotiation.cc:299] Negotiation successful
0925 04:37:08.984287 (+ 1002us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
W20260925 04:37:08.986207 12304 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
W20260925 04:37:08.986359 12305 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
I20260925 04:37:08.986464 12305 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b; no voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af
I20260925 04:37:08.986687 12695 raft_consensus.cc:2726] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:37:09.061969 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:09.061595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:55725)
0925 04:37:09.061822 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:09.061870 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
I20260925 04:37:09.711527 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:09.711117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:53835)
0925 04:37:09.711354 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:09.711429 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260925 04:37:09.856722 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:09.856368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:43117)
0925 04:37:09.856589 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:09.856637 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260925 04:37:10.062937 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:10.062572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:54015)
0925 04:37:10.062782 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:10.062839 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260925 04:37:10.712699 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:10.712285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:60975)
0925 04:37:10.712545 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:10.712601 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260925 04:37:10.857777 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:10.857365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:57925)
0925 04:37:10.857635 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:10.857683 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":64,"threads_started":1}
I20260925 04:37:10.924374 12705 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:10.924505 12705 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:10.924975 12705 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
W20260925 04:37:10.925894 12304 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
W20260925 04:37:10.926000 12305 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
I20260925 04:37:10.926059 12305 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b; no voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af
I20260925 04:37:10.926204 12705 raft_consensus.cc:2726] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:37:11.064041 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:11.063645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:38223)
0925 04:37:11.063908 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:11.063957 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260925 04:37:11.713982 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:11.713479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:42235)
0925 04:37:11.713830 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:11.713887 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":93,"threads_started":1}
I20260925 04:37:11.858966 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:11.858525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:33571)
0925 04:37:11.858804 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:11.858872 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260925 04:37:12.065184 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:12.064772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:52715)
0925 04:37:12.065037 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:12.065091 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260925 04:37:12.715194 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:12.714800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:40115)
0925 04:37:12.715052 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:12.715100 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":68,"threads_started":1}
I20260925 04:37:12.860244 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:12.859806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:44703)
0925 04:37:12.860070 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:12.860129 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260925 04:37:13.066619 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:13.066021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:53703)
0925 04:37:13.066291 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:13.066350 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260925 04:37:13.716439 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:13.715965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:56959)
0925 04:37:13.716253 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:13.716319 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
I20260925 04:37:13.861399 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:13.860981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:45147)
0925 04:37:13.861239 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:13.861306 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260925 04:37:13.975742 12715 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:13.975900 12715 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:13.976431 12715 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
W20260925 04:37:13.977144 12305 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
W20260925 04:37:13.977277 12304 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
I20260925 04:37:13.977375 12304 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b; no voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af
I20260925 04:37:13.977576 12715 raft_consensus.cc:2726] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:37:14.067849 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:14.067378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:46875)
0925 04:37:14.067680 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:14.067741 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260925 04:37:14.717733 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:14.717270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:52905)
0925 04:37:14.717586 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:14.717638 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260925 04:37:14.862474 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:14.862075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:54967)
0925 04:37:14.862339 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:14.862389 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260925 04:37:15.069144 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:15.068638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:59011)
0925 04:37:15.069005 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:15.069057 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":61,"threads_started":1}
I20260925 04:37:15.485091 12720 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:15.485265 12720 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:15.485824 12720 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
W20260925 04:37:15.486625 12304 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
W20260925 04:37:15.486711 12305 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
I20260925 04:37:15.486756 12304 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b; no voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af
I20260925 04:37:15.486920 12720 raft_consensus.cc:2726] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:37:15.718760 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:15.718369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:48503)
0925 04:37:15.718612 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:15.718665 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260925 04:37:15.863747 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:15.863201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:37349)
0925 04:37:15.863590 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:15.863651 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":82,"threads_started":1}
I20260925 04:37:16.070317 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:16.069911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:49363)
0925 04:37:16.070160 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:16.070215 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260925 04:37:16.720010 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:16.719449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:44735)
0925 04:37:16.719820 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:16.719885 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":82,"threads_started":1}
I20260925 04:37:16.864898 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:16.864489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:49889)
0925 04:37:16.864755 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:16.864809 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
I20260925 04:37:17.071452 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:17.070992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:59427)
0925 04:37:17.071293 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:17.071354 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20260925 04:37:17.721398 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:17.720932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:46893)
0925 04:37:17.721235 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:17.721297 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260925 04:37:17.866142 12728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:17.865692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:38797)
0925 04:37:17.865964 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:17.866030 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260925 04:37:18.072654 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:18.072250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:48963)
0925 04:37:18.072511 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:18.072562 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260925 04:37:18.722548 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:18.722159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:50915)
0925 04:37:18.722398 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:18.722452 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260925 04:37:18.867313 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:18.866921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:45649)
0925 04:37:18.867172 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:18.867224 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260925 04:37:19.073856 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:19.073375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:32913)
0925 04:37:19.073688 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:19.073742 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
I20260925 04:37:19.723913 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:19.723448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:46205)
0925 04:37:19.723729 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:19.723797 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260925 04:37:19.868438 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:19.868025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:50383)
0925 04:37:19.868284 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:19.868341 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260925 04:37:20.075027 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:20.074559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:53859)
0925 04:37:20.074876 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:20.074931 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
I20260925 04:37:20.725230 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:20.724835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:53773)
0925 04:37:20.725069 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:20.725132 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260925 04:37:20.869767 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:20.869239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:60961)
0925 04:37:20.869597 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:20.869667 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260925 04:37:21.076277 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:21.075861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:58117)
0925 04:37:21.076124 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:21.076181 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":87,"threads_started":1}
I20260925 04:37:21.726341 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:21.725923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:36775)
0925 04:37:21.726173 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:21.726229 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260925 04:37:21.871027 12740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:21.870585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:38431)
0925 04:37:21.870850 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:21.870913 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260925 04:37:22.077698 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:22.077206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.195:33731)
0925 04:37:22.077481 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:22.077596 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260925 04:37:22.727666 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:22.727268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.193:47363)
0925 04:37:22.727518 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:22.727571 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260925 04:37:22.766407 12743 raft_consensus.cc:482] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:22.766561 12743 raft_consensus.cc:504] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83947d33083d403395a91ab4db50945d" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39221 } } peers { permanent_uuid: "d92bd3ecf5e24096bdf409fb6cd794af" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 38071 } } peers { permanent_uuid: "2209b5979dc24803ada11e45a668a61b" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 35007 } }
I20260925 04:37:22.767045 12743 leader_election.cc:288] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83947d33083d403395a91ab4db50945d (127.27.46.195:39221), d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071)
W20260925 04:37:22.767781 12305 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83947d33083d403395a91ab4db50945d (127.27.46.195:39221): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
W20260925 04:37:22.767881 12304 leader_election.cc:341] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d92bd3ecf5e24096bdf409fb6cd794af (127.27.46.194:38071): Not found: Tablet not found: 356c661f67254fb1b66cfee6d980dada
I20260925 04:37:22.767948 12304 leader_election.cc:302] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2209b5979dc24803ada11e45a668a61b; no voters: 83947d33083d403395a91ab4db50945d, d92bd3ecf5e24096bdf409fb6cd794af
I20260925 04:37:22.768105 12743 raft_consensus.cc:2726] T 356c661f67254fb1b66cfee6d980dada P 2209b5979dc24803ada11e45a668a61b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:37:22.872335 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:22.871931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38215 (local address 127.27.46.194:54351)
0925 04:37:22.872171 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:22.872220 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.254:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260925 04:37:23.064361 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12247
I20260925 04:37:23.070863 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12424
I20260925 04:37:23.076611 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12559
2026-09-25T04:37:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59721 ms)
[----------] 30 tests from SecurityITest (240993 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:23 dist-test-slave-r8r3 krb5kdc[12747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:23 dist-test-slave-r8r3 krb5kdc[12747](info): set up 2 sockets
Sep 25 04:37:23 dist-test-slave-r8r3 krb5kdc[12747](info): commencing operation
krb5kdc: starting...
W20260925 04:37:24.105787 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:24.228802 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41235
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:41235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:24.339877 12760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:24.340163 12760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:24.340246 12760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:24.344259 12760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:24.344341 12760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:24.344380 12760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:24.344439 12760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:24.344478 12760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:24.349690 12760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:41235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:24.350714 12760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:24.351697 12760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:24.360356 12765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:24.360545 12768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:24.360446 12760 server_base.cc:1034] running on GCE node
W20260925 04:37:24.360496 12766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:24.361202 12760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:24.362313 12760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:24.363525 12760 hybrid_clock.cc:648] HybridClock initialized: now 1790311044363508 us; error 34 us; skew 500 ppm
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:24.366176 12760 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:24.367005 12760 webserver.cc:460] Webserver started at http://127.27.46.254:41635/ using document root <none> and password file <none>
I20260925 04:37:24.367533 12760 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:24.367619 12760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:24.367856 12760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:24.369848 12760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "15518272d1f24d90ba6c4701690abcb8"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "1844e3c1987c0fbde9253710ac1c650c"
server_key_iv: "e6dc2ef406db83a93be86c6c5b794da9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.370446 12760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "15518272d1f24d90ba6c4701690abcb8"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "1844e3c1987c0fbde9253710ac1c650c"
server_key_iv: "e6dc2ef406db83a93be86c6c5b794da9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.374063 12760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:24.376402 12775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.377697 12760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:24.377842 12760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "15518272d1f24d90ba6c4701690abcb8"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "1844e3c1987c0fbde9253710ac1c650c"
server_key_iv: "e6dc2ef406db83a93be86c6c5b794da9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.377954 12760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:24.394997 12760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:24.397823 12760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:24.398021 12760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:24.405262 12760 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:41235
I20260925 04:37:24.405287 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:41235 every 8 connection(s)
I20260925 04:37:24.406170 12760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:24.408756 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:24.414083 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: Bootstrap starting.
I20260925 04:37:24.414624 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12760
I20260925 04:37:24.414765 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:24.415077 27835 external_mini_cluster.cc:1428] Setting key 326ec9ebb2562597c30f1d3a86364f26
I20260925 04:37:24.417003 12829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:24.417798 12829 log.cc:824] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:24.420073 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: No bootstrap required, opened a new log
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:24.424712 12829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } }
I20260925 04:37:24.424924 12829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:24.424989 12829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15518272d1f24d90ba6c4701690abcb8, State: Initialized, Role: FOLLOWER
I20260925 04:37:24.425539 12829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } }
I20260925 04:37:24.425717 12829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:24.425802 12829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:24.425918 12829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:24.427136 12829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } }
I20260925 04:37:24.427517 12829 leader_election.cc:302] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15518272d1f24d90ba6c4701690abcb8; no voters:
I20260925 04:37:24.427863 12829 leader_election.cc:288] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:24.427960 12834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:24.428225 12834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [term 1 LEADER]: Becoming Leader. State: Replica: 15518272d1f24d90ba6c4701690abcb8, State: Running, Role: LEADER
I20260925 04:37:24.428558 12834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } }
I20260925 04:37:24.429224 12829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:24.430053 12835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15518272d1f24d90ba6c4701690abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } } }
I20260925 04:37:24.430164 12835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:24.430388 12835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15518272d1f24d90ba6c4701690abcb8. Latest consensus state: current_term: 1 leader_uuid: "15518272d1f24d90ba6c4701690abcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15518272d1f24d90ba6c4701690abcb8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 41235 } } }
I20260925 04:37:24.430461 12835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:24.431170 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:24.416472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.27.46.254:41235)
0925 04:37:24.416921 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:24.416936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:24.416970 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:37:24.417649 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:24.417666 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:24.418047 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:24.418446 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:24.419374 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.420484 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.421719 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.422205 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.422340 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:24.425173 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:24.425179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:24.425194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:24.425226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:24.427364 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.427819 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.427827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.427834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.427980 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.428351 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.428354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.428356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.428833 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:24.429595 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:24.429769 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 04:37:24.430093 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":125,"threads_started":1}
W20260925 04:37:24.434296 12849 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:37:24.434378 12849 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:37:24.434468 12843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:24.435343 12843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:24.440258 12843 catalog_manager.cc:1347] Generated new cluster ID: 7cd2fe2bd2ea4d1684fc22a7cd639b87
I20260925 04:37:24.440320 12843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:24.466487 12843 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:24.467742 12843 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:24.479247 12843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: Generated new TSK 0
I20260925 04:37:24.480073 12843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:37:24.512719 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:24.621364 12855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:24.621600 12855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:24.621673 12855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:24.625348 12855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:24.625396 12855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:24.625443 12855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:24.630862 12855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:24.631932 12855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:24.632933 12855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:24.642421 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:24.642475 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:24.642421 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:24.642514 12855 server_base.cc:1034] running on GCE node
I20260925 04:37:24.643060 12855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:24.643683 12855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:24.644876 12855 hybrid_clock.cc:648] HybridClock initialized: now 1790311044644857 us; error 31 us; skew 500 ppm
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:24.647428 12855 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:37:24.648372 12855 webserver.cc:460] Webserver started at http://127.27.46.193:38113/ using document root <none> and password file <none>
I20260925 04:37:24.648926 12855 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:24.649016 12855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:24.649266 12855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:24.651275 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "fb65946988b84350904f8ff6d747fe23"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "a2e69bf3c597b0d57343c32e164e8fba"
server_key_iv: "5c50e8bcb319e5fe3a4f949ac9adcea9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.651839 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "fb65946988b84350904f8ff6d747fe23"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "a2e69bf3c597b0d57343c32e164e8fba"
server_key_iv: "5c50e8bcb319e5fe3a4f949ac9adcea9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.655309 12855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:37:24.657614 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.658738 12855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:24.658874 12855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "fb65946988b84350904f8ff6d747fe23"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "a2e69bf3c597b0d57343c32e164e8fba"
server_key_iv: "5c50e8bcb319e5fe3a4f949ac9adcea9"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.658973 12855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:24.689893 12855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:24.692692 12855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:24.692912 12855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:24.693595 12855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:24.694459 12855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:24.694531 12855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.694604 12855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:24.694664 12855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.703740 12855 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:46335
I20260925 04:37:24.703775 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:46335 every 8 connection(s)
I20260925 04:37:24.704594 12855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:24.708609 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12855
I20260925 04:37:24.708760 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:24.709100 27835 external_mini_cluster.cc:1428] Setting key 88ccb1d9efbd9aff5969e9043c64a590
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:37:24.718856 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:24.706469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:49273 (local address 127.27.46.254:41235)
0925 04:37:24.706638 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:24.706643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:24.707417 (+ 774us) server_negotiation.cc:407] Connection header received
0925 04:37:24.708704 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:24.708709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:24.708785 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:24.708879 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:24.710188 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.710844 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.711997 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.712310 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.712394 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:24.715274 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:24.715283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:24.715285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:24.715309 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:24.716747 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.717268 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.717273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.717277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.717360 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.717741 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.717746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.717747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.717935 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:24.718048 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:24.718549 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 04:37:24.718671 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:37:24.719729 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:24.706832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41235 (local address 127.27.46.193:49273)
0925 04:37:24.707272 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:24.707315 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:24.708495 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:24.709059 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:24.709065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:24.709465 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:24.710048 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:24.710062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.710973 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.710976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:24.711864 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:24.711874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.712479 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.712485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:24.712716 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:24.713304 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:24.713330 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:24.715108 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:24.716878 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:24.716884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:24.716898 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:24.717160 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:24.717465 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:24.717468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:24.717470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:24.717641 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:24.718047 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:24.718054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:24.718309 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 04:37:24.718491 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:37:24.718810 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260925 04:37:24.720993 12984 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:37:24.721279 12984 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:24.721843 12984 heartbeater.cc:511] Master 127.27.46.254:41235 requested a full tablet report, sending...
I20260925 04:37:24.723356 12792 ts_manager.cc:194] Registered new tserver with Master: fb65946988b84350904f8ff6d747fe23 (127.27.46.193:46335)
I20260925 04:37:24.725010 12792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:49273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:37:24.736771 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:24.846019 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:24.846266 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:24.846349 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:24.850428 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:24.850485 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:24.850602 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:24.855783 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:24.856770 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:24.857879 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:24.867630 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:24.867816 12989 server_base.cc:1034] running on GCE node
W20260925 04:37:24.867621 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:24.867623 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:24.868454 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:24.869021 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:24.870184 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790311044870171 us; error 25 us; skew 500 ppm
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:24.872882 12989 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:37:24.873757 12989 webserver.cc:460] Webserver started at http://127.27.46.194:33019/ using document root <none> and password file <none>
I20260925 04:37:24.874305 12989 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:24.874365 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:24.874611 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:24.876588 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "e31b95f231c74f1f9a530d4b0ec897bc"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "9322bc92d39babfd5696f1bb8a6f8401"
server_key_iv: "8fcf07fa7a31bb06c95d35041824a78e"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.877137 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "e31b95f231c74f1f9a530d4b0ec897bc"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "9322bc92d39babfd5696f1bb8a6f8401"
server_key_iv: "8fcf07fa7a31bb06c95d35041824a78e"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.880676 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:37:24.883152 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.884366 12989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:37:24.884483 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "e31b95f231c74f1f9a530d4b0ec897bc"
format_stamp: "Formatted at 2026-09-25 04:37:24 on dist-test-slave-r8r3"
server_key: "9322bc92d39babfd5696f1bb8a6f8401"
server_key_iv: "8fcf07fa7a31bb06c95d35041824a78e"
server_key_version: "encryptionkey@0"
I20260925 04:37:24.884584 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:24.916606 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:24.919981 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:24.920192 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:24.920794 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:24.921630 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:24.921689 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.921801 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:24.921839 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:24.931246 12989 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:34403
I20260925 04:37:24.931300 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:34403 every 8 connection(s)
I20260925 04:37:24.932276 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:24.933287 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 12989
I20260925 04:37:24.933404 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:24.933772 27835 external_mini_cluster.cc:1428] Setting key b90896b8f9b181d77cbcdb91a045ae2b
Sep 25 04:37:24 dist-test-slave-r8r3 krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311044, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:37:24.947057 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:24.934307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:48575 (local address 127.27.46.254:41235)
0925 04:37:24.934427 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:24.934432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:24.935409 (+ 977us) server_negotiation.cc:407] Connection header received
0925 04:37:24.937428 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:24.937433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:24.937516 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:24.937613 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:24.938759 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.939314 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.940581 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.940802 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.940912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:24.943533 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:24.943537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:24.943539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:24.943563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:24.944971 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.945469 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.945473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.945475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.945597 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:24.945927 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:24.945932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:24.945933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:24.946109 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:24.946224 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:24.946762 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 04:37:24.946875 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 04:37:24.947863 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:24.934743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41235 (local address 127.27.46.194:48575)
0925 04:37:24.935270 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:24.935324 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:24.937171 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:24.937794 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:24.937801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:24.938153 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:24.938626 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:24.938640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.939467 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.939470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:24.940456 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:24.940466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:24.940953 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:24.940958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:24.941124 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:24.941727 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:24.941742 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:24.943365 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:24.945101 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:24.945106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:24.945120 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:24.945367 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:24.945693 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:24.945696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:24.945698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:24.945833 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:24.946221 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:24.946229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:24.946509 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 04:37:24.946718 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 04:37:24.947045 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260925 04:37:24.949098 13123 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41235
I20260925 04:37:24.949390 13123 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:24.950021 13123 heartbeater.cc:511] Master 127.27.46.254:41235 requested a full tablet report, sending...
I20260925 04:37:24.951303 12792 ts_manager.cc:194] Registered new tserver with Master: e31b95f231c74f1f9a530d4b0ec897bc (127.27.46.194:34403)
I20260925 04:37:24.952209 12792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:48575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:37:24.964314 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:25.077204 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:25.077455 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:25.077574 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:25.081471 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:25.081578 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:25.081645 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:25.086859 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:41235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:25.087874 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:25.088933 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:25.098358 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.098344 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.098439 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:25.098876 13128 server_base.cc:1034] running on GCE node
I20260925 04:37:25.099279 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:25.099949 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:25.101171 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790311045101162 us; error 30 us; skew 500 ppm
Sep 25 04:37:25 dist-test-slave-r8r3 krb5kdc[12747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:25.103956 13128 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:37:25.104815 13128 webserver.cc:460] Webserver started at http://127.27.46.195:45639/ using document root <none> and password file <none>
I20260925 04:37:25.105296 13128 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:25.105350 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:25.105569 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:25.107465 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "b25cebaa34874acebfd91e74ba40ae4c"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "9baf1be46fba885d4090af1e7f7dc660"
server_key_iv: "fa9efd7910ece160585a92a9c8d482f7"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.107982 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "b25cebaa34874acebfd91e74ba40ae4c"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "9baf1be46fba885d4090af1e7f7dc660"
server_key_iv: "fa9efd7910ece160585a92a9c8d482f7"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.111588 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:37:25.113991 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.115168 13128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:25.115257 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "b25cebaa34874acebfd91e74ba40ae4c"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "9baf1be46fba885d4090af1e7f7dc660"
server_key_iv: "fa9efd7910ece160585a92a9c8d482f7"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.115335 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:25.141778 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:25.144757 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:25.144919 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:25.145464 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:25.146344 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:25.146392 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.146433 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:25.146449 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.155570 13128 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:35427
I20260925 04:37:25.155599 13258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:35427 every 8 connection(s)
I20260925 04:37:25.156445 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:25.160787 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13128
I20260925 04:37:25.160940 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:25.161296 27835 external_mini_cluster.cc:1428] Setting key b18531ce4590a2776aba85345557ec4a
Sep 25 04:37:25 dist-test-slave-r8r3 krb5kdc[12747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311045, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:25.170616 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.158429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:42495 (local address 127.27.46.254:41235)
0925 04:37:25.158565 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.158569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.159371 (+ 802us) server_negotiation.cc:407] Connection header received
0925 04:37:25.160622 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.160625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.160672 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.160750 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:25.162318 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.163000 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.164468 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.164683 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.164771 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.167182 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:25.167184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:25.167186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:25.167214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:25.168759 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:25.169227 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:25.169230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:25.169231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:25.169304 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:25.169681 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:25.169683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:25.169684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:25.169862 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:25.169938 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.170404 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.170494 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:37:25.171452 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.158738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:41235 (local address 127.27.46.195:42495)
0925 04:37:25.159206 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.159248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.160395 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.160954 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.160964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.161435 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:25.162044 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.162063 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.163121 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.163125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.164354 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.164364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.164822 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.164825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.164960 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.165463 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:25.165478 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:25.167030 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:25.168870 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:25.168875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:25.168887 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:25.169138 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:25.169384 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:25.169386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:25.169388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:25.169592 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:25.169952 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:25.169957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:25.170193 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.170343 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.170587 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260925 04:37:25.172605 13259 heartbeater.cc:347] Connected to a master server at 127.27.46.254:41235
I20260925 04:37:25.172843 13259 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:25.173300 13259 heartbeater.cc:511] Master 127.27.46.254:41235 requested a full tablet report, sending...
I20260925 04:37:25.174363 12792 ts_manager.cc:194] Registered new tserver with Master: b25cebaa34874acebfd91e74ba40ae4c (127.27.46.195:35427)
I20260925 04:37:25.174975 12792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:42495
I20260925 04:37:25.175343 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:25.184574 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.177617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.27.46.254:41235)
0925 04:37:25.177758 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.177762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.177944 (+ 182us) server_negotiation.cc:407] Connection header received
0925 04:37:25.178034 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.178036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.178073 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.178135 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:25.178775 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.179235 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.180046 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.180246 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.180318 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.181590 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:25.181593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:25.181595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:25.181616 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:25.183047 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:25.183493 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:25.183496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:25.183497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:25.183577 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:25.183821 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:25.183824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:25.183825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:25.184024 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:25.184138 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.184291 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.184383 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:37:25.187316 12792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:25.190469 12792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:25.205003 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.199009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.27.46.194:34403)
0925 04:37:25.199450 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.199463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.199491 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:37:25.199780 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.199786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.200007 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.200138 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:25.200647 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.201846 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.203677 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204262 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204384 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.204534 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.204743 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.204850 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":108,"threads_started":1}
I20260925 04:37:25.205163 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.199058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:46335 (local address 127.0.0.1:39660)
0925 04:37:25.199674 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.199687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.199778 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.200892 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.200895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.200913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:25.201021 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.201030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.202229 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.202232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.203574 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.203584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204703 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.204789 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.204811 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.204870 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.204985 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":147,"thread_start_us":94,"threads_started":1}
I20260925 04:37:25.205174 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.199670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.27.46.193:46335)
0925 04:37:25.200454 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.200471 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.200498 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:37:25.200575 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.200580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.200755 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.200915 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:25.201180 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.202094 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.203716 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204571 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204690 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.204858 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.204911 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.205005 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":124,"threads_started":1}
I20260925 04:37:25.205380 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.199792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:35427 (local address 127.0.0.1:35698)
0925 04:37:25.200046 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.200058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.200153 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.201138 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.201144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.201166 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:25.201251 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.201257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.202477 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.202480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.203769 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.203779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204961 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.205055 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.205090 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.205158 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.205246 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":104,"thread_start_us":102,"threads_started":1}
I20260925 04:37:25.205852 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.200347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.27.46.195:35427)
0925 04:37:25.200712 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.200724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.200745 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:37:25.200815 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.200820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.200995 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.201151 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:25.201349 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.202313 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.203900 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204701 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.204900 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.205635 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.205729 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":143,"threads_started":1}
I20260925 04:37:25.206087 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.198861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:34403 (local address 127.0.0.1:47726)
0925 04:37:25.199371 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.199422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.199558 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.200304 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.200309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.200334 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:25.200475 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.200484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.201981 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.201984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.203547 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.203558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.204396 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.204400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.204505 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.204542 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.205878 (+ 1336us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.205966 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":90,"thread_start_us":79,"threads_started":1}
I20260925 04:37:25.208500 13055 tablet_service.cc:1467] Processing CreateTablet for tablet d4c3cdff123a4dc388177c9ff2681127 (DEFAULT_TABLE table=test-table [id=e6ebea748b3d4be6892f57abed1367d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:25.208642 12918 tablet_service.cc:1467] Processing CreateTablet for tablet d4c3cdff123a4dc388177c9ff2681127 (DEFAULT_TABLE table=test-table [id=e6ebea748b3d4be6892f57abed1367d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:25.208642 13191 tablet_service.cc:1467] Processing CreateTablet for tablet d4c3cdff123a4dc388177c9ff2681127 (DEFAULT_TABLE table=test-table [id=e6ebea748b3d4be6892f57abed1367d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:25.209704 12918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c3cdff123a4dc388177c9ff2681127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:25.209709 13191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c3cdff123a4dc388177c9ff2681127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:25.209702 13055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c3cdff123a4dc388177c9ff2681127. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:25.215968 13276 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Bootstrap starting.
I20260925 04:37:25.217483 13277 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Bootstrap starting.
I20260925 04:37:25.217844 13278 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Bootstrap starting.
I20260925 04:37:25.218321 13276 tablet_bootstrap.cc:654] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:25.219702 13276 log.cc:824] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:25.219816 13278 tablet_bootstrap.cc:654] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:25.219816 13277 tablet_bootstrap.cc:654] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:25.221135 13277 log.cc:824] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:25.221135 13278 log.cc:824] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:25.221779 13276 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: No bootstrap required, opened a new log
I20260925 04:37:25.221982 13276 ts_tablet_manager.cc:1397] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:37:25.222839 13277 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: No bootstrap required, opened a new log
I20260925 04:37:25.223042 13277 ts_tablet_manager.cc:1397] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 04:37:25.223274 13278 tablet_bootstrap.cc:492] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: No bootstrap required, opened a new log
I20260925 04:37:25.223474 13278 ts_tablet_manager.cc:1397] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:37:25.225919 13276 raft_consensus.cc:348] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.226202 13276 raft_consensus.cc:374] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:25.226282 13276 raft_consensus.cc:729] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31b95f231c74f1f9a530d4b0ec897bc, State: Initialized, Role: FOLLOWER
I20260925 04:37:25.226275 13277 raft_consensus.cc:348] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.226541 13277 raft_consensus.cc:374] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:25.226624 13277 raft_consensus.cc:729] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25cebaa34874acebfd91e74ba40ae4c, State: Initialized, Role: FOLLOWER
I20260925 04:37:25.226735 13276 consensus_queue.cc:260] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.227089 13277 consensus_queue.cc:260] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.227138 13278 raft_consensus.cc:348] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.227483 13123 heartbeater.cc:503] Master 127.27.46.254:41235 was elected leader, sending a full tablet report...
I20260925 04:37:25.227746 13259 heartbeater.cc:503] Master 127.27.46.254:41235 was elected leader, sending a full tablet report...
I20260925 04:37:25.227998 13277 ts_tablet_manager.cc:1428] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:37:25.228420 13278 raft_consensus.cc:374] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:25.228492 13278 raft_consensus.cc:729] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb65946988b84350904f8ff6d747fe23, State: Initialized, Role: FOLLOWER
I20260925 04:37:25.228843 13276 ts_tablet_manager.cc:1428] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:37:25.228997 13278 consensus_queue.cc:260] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.230019 13278 ts_tablet_manager.cc:1428] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:37:25.230800 12984 heartbeater.cc:503] Master 127.27.46.254:41235 was elected leader, sending a full tablet report...
I20260925 04:37:25.374612 13283 raft_consensus.cc:482] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:25.374835 13283 raft_consensus.cc:504] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.376288 13283 leader_election.cc:288] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e31b95f231c74f1f9a530d4b0ec897bc (127.27.46.194:34403), fb65946988b84350904f8ff6d747fe23 (127.27.46.193:46335)
I20260925 04:37:25.380643 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.376679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:42671 (local address 127.27.46.193:46335)
0925 04:37:25.376803 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.376807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.376825 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:37:25.376985 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.376987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.377033 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.377100 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:25.377455 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.378176 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.379496 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.380008 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.380103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.380145 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.380415 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.380481 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:37:25.380656 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.376592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:46335 (local address 127.27.46.195:42671)
0925 04:37:25.376722 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.376741 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.376857 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.377153 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.377157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.377178 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:25.377316 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.377323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.378295 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.378299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.379362 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.379372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.380132 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.380136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.380253 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.380281 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.380371 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.380450 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260925 04:37:25.381047 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.376686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:56325 (local address 127.27.46.194:34403)
0925 04:37:25.376857 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.376863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.377456 (+ 593us) server_negotiation.cc:407] Connection header received
0925 04:37:25.377592 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.377594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.377644 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.377709 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:25.378003 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.378718 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.379844 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.380462 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.380582 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.380628 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.380862 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.380935 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:37:25.381095 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.376596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:34403 (local address 127.27.46.195:56325)
0925 04:37:25.377325 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.377336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.377454 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.377748 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.377751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.377770 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:25.377888 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.377893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.378830 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.378833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.379712 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.379722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.380582 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.380585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.380703 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.380729 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.380855 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.380926 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":345,"thread_start_us":181,"threads_started":1}
I20260925 04:37:25.381428 12938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c3cdff123a4dc388177c9ff2681127" candidate_uuid: "b25cebaa34874acebfd91e74ba40ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65946988b84350904f8ff6d747fe23" is_pre_election: true
I20260925 04:37:25.381851 12938 raft_consensus.cc:2454] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25cebaa34874acebfd91e74ba40ae4c in term 0.
I20260925 04:37:25.381846 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c3cdff123a4dc388177c9ff2681127" candidate_uuid: "b25cebaa34874acebfd91e74ba40ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" is_pre_election: true
I20260925 04:37:25.382129 13075 raft_consensus.cc:2454] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25cebaa34874acebfd91e74ba40ae4c in term 0.
I20260925 04:37:25.382437 13146 leader_election.cc:302] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25cebaa34874acebfd91e74ba40ae4c, fb65946988b84350904f8ff6d747fe23; no voters:
I20260925 04:37:25.382776 13283 raft_consensus.cc:2781] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:25.382865 13283 raft_consensus.cc:482] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:25.382908 13283 raft_consensus.cc:3037] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:25.383950 13283 raft_consensus.cc:504] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.384332 13283 leader_election.cc:288] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [CANDIDATE]: Term 1 election: Requested vote from peers e31b95f231c74f1f9a530d4b0ec897bc (127.27.46.194:34403), fb65946988b84350904f8ff6d747fe23 (127.27.46.193:46335)
I20260925 04:37:25.384621 13075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c3cdff123a4dc388177c9ff2681127" candidate_uuid: "b25cebaa34874acebfd91e74ba40ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31b95f231c74f1f9a530d4b0ec897bc"
I20260925 04:37:25.384790 12938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c3cdff123a4dc388177c9ff2681127" candidate_uuid: "b25cebaa34874acebfd91e74ba40ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65946988b84350904f8ff6d747fe23"
I20260925 04:37:25.384809 13075 raft_consensus.cc:3037] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:25.384922 12938 raft_consensus.cc:3037] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:25.386198 12938 raft_consensus.cc:2454] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25cebaa34874acebfd91e74ba40ae4c in term 1.
I20260925 04:37:25.386198 13075 raft_consensus.cc:2454] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25cebaa34874acebfd91e74ba40ae4c in term 1.
I20260925 04:37:25.386534 13144 leader_election.cc:302] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25cebaa34874acebfd91e74ba40ae4c, e31b95f231c74f1f9a530d4b0ec897bc; no voters:
I20260925 04:37:25.386718 13283 raft_consensus.cc:2781] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:25.386962 13283 raft_consensus.cc:686] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 1 LEADER]: Becoming Leader. State: Replica: b25cebaa34874acebfd91e74ba40ae4c, State: Running, Role: LEADER
I20260925 04:37:25.387315 13283 consensus_queue.cc:237] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } }
I20260925 04:37:25.390827 12792 catalog_manager.cc:5495] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c reported cstate change: term changed from 0 to 1, leader changed from <none> to b25cebaa34874acebfd91e74ba40ae4c (127.27.46.195). New cstate: current_term: 1 leader_uuid: "b25cebaa34874acebfd91e74ba40ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25cebaa34874acebfd91e74ba40ae4c" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 35427 } health_report { overall_health: HEALTHY } } }
W20260925 04:37:25.407477 13260 tablet.cc:2369] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:37:25.433513 13124 tablet.cc:2369] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:25.443169 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.439589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.27.46.195:35427)
0925 04:37:25.439787 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.439792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.439813 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:37:25.439884 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.439887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.439941 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.440053 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:25.440323 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.440832 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.441841 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.442044 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.442122 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.442385 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:25.442787 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:25.442873 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.442928 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.442977 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 04:37:25.448968 13075 raft_consensus.cc:1261] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 1 FOLLOWER]: Refusing update from remote peer b25cebaa34874acebfd91e74ba40ae4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:25.448966 12938 raft_consensus.cc:1261] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 1 FOLLOWER]: Refusing update from remote peer b25cebaa34874acebfd91e74ba40ae4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:25.449724 13283 consensus_queue.cc:1035] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31b95f231c74f1f9a530d4b0ec897bc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 34403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:25.449945 13286 consensus_queue.cc:1035] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb65946988b84350904f8ff6d747fe23" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:37:25.456080 12985 tablet.cc:2369] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:25.456487 13289 mvcc.cc:205] Tried to move back new op lower bound from 7333114042151600128 to 7333114041909534720. Current Snapshot: MvccSnapshot[applied={T|T < 7333114042151600128}]
I20260925 04:37:25.456938 13291 mvcc.cc:205] Tried to move back new op lower bound from 7333114042151600128 to 7333114041909534720. Current Snapshot: MvccSnapshot[applied={T|T < 7333114042151600128}]
I20260925 04:37:25.459478 13290 mvcc.cc:205] Tried to move back new op lower bound from 7333114042151600128 to 7333114041909534720. Current Snapshot: MvccSnapshot[applied={T|T < 7333114042151600128}]
I20260925 04:37:25.465912 12790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:25.466104 12790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:25.468506 12790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15518272d1f24d90ba6c4701690abcb8: Sending DeleteTablet for 3 replicas of tablet d4c3cdff123a4dc388177c9ff2681127
I20260925 04:37:25.469262 13191 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c3cdff123a4dc388177c9ff2681127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:35698
I20260925 04:37:25.469240 13055 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c3cdff123a4dc388177c9ff2681127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47726
I20260925 04:37:25.469800 13301 tablet_replica.cc:330] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: stopping tablet replica
I20260925 04:37:25.469830 12918 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c3cdff123a4dc388177c9ff2681127 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:25 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:39660
I20260925 04:37:25.470034 13300 tablet_replica.cc:330] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: stopping tablet replica
I20260925 04:37:25.470372 13300 raft_consensus.cc:2229] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:37:25.470697 13302 tablet_replica.cc:330] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23: stopping tablet replica
I20260925 04:37:25.470779 13300 raft_consensus.cc:2258] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:25.470803 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12855
I20260925 04:37:25.471036 13302 raft_consensus.cc:2229] T d4c3cdff123a4dc388177c9ff2681127 P fb65946988b84350904f8ff6d747fe23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:25.471107 13301 raft_consensus.cc:2229] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:25.471439 13301 raft_consensus.cc:2258] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:25.473198 13301 ts_tablet_manager.cc:1905] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:25.476054 13300 ts_tablet_manager.cc:1905] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:25.476100 13301 ts_tablet_manager.cc:1918] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:25.476298 13301 log.cc:1196] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/d4c3cdff123a4dc388177c9ff2681127
I20260925 04:37:25.476701 13301 ts_tablet_manager.cc:1939] T d4c3cdff123a4dc388177c9ff2681127 P e31b95f231c74f1f9a530d4b0ec897bc: Deleting consensus metadata
I20260925 04:37:25.477998 12776 catalog_manager.cc:4841] TS e31b95f231c74f1f9a530d4b0ec897bc (127.27.46.194:34403): tablet d4c3cdff123a4dc388177c9ff2681127 (table test-table [id=e6ebea748b3d4be6892f57abed1367d8]) successfully deleted
I20260925 04:37:25.479559 13300 ts_tablet_manager.cc:1918] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:25.479663 13300 log.cc:1196] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/d4c3cdff123a4dc388177c9ff2681127
I20260925 04:37:25.480029 13300 ts_tablet_manager.cc:1939] T d4c3cdff123a4dc388177c9ff2681127 P b25cebaa34874acebfd91e74ba40ae4c: Deleting consensus metadata
W20260925 04:37:25.480223 12778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:46335 (error 108)
I20260925 04:37:25.480430 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12989
I20260925 04:37:25.482254 12777 catalog_manager.cc:4841] TS b25cebaa34874acebfd91e74ba40ae4c (127.27.46.195:35427): tablet d4c3cdff123a4dc388177c9ff2681127 (table test-table [id=e6ebea748b3d4be6892f57abed1367d8]) successfully deleted
I20260925 04:37:25.482455 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.482117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:46335 (local address 127.0.0.1:39066)
0925 04:37:25.482302 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.482369 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:46335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 04:37:25.482617 12778 catalog_manager.cc:4584] TS fb65946988b84350904f8ff6d747fe23 (127.27.46.193:46335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4c3cdff123a4dc388177c9ff2681127: Network error: Client connection negotiation failed: client connection to 127.27.46.193:46335: connect: Connection refused (error 111)
I20260925 04:37:25.486975 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13128
I20260925 04:37:25.493610 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 12760
2026-09-25T04:37:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2416 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T04:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:25Z Disabled control of system clock
I20260925 04:37:25.524464 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:36711
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:25.631263 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:25.631518 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:25.631552 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:25.635416 13307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:25.635462 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:25.635478 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:25.635492 13307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:25.635506 13307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:25.640064 13307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:36711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:25.641160 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:25.642659 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:25.650986 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.650997 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.651016 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:25.651374 13307 server_base.cc:1034] running on GCE node
I20260925 04:37:25.651921 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:25.652966 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:25.654117 13307 hybrid_clock.cc:648] HybridClock initialized: now 1790311045654088 us; error 42 us; skew 500 ppm
I20260925 04:37:25.655875 13307 webserver.cc:460] Webserver started at http://127.27.46.254:39893/ using document root <none> and password file <none>
I20260925 04:37:25.656400 13307 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:25.656457 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:25.656610 13307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:25.658619 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "1eed85d8dc9d4f8eae960b1d60a61e56"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "062c5b6182ddfe5eec76d95b983c48e9"
server_key_iv: "58df42f13fa3399f82db692593a1fb69"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.659118 13307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "1eed85d8dc9d4f8eae960b1d60a61e56"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "062c5b6182ddfe5eec76d95b983c48e9"
server_key_iv: "58df42f13fa3399f82db692593a1fb69"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.662520 13307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:37:25.664835 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.666118 13307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:25.666217 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "1eed85d8dc9d4f8eae960b1d60a61e56"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "062c5b6182ddfe5eec76d95b983c48e9"
server_key_iv: "58df42f13fa3399f82db692593a1fb69"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.666320 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:25.692546 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:25.693197 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:25.693351 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:25.700479 13307 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:36711
I20260925 04:37:25.700568 13374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:36711 every 8 connection(s)
I20260925 04:37:25.701483 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:25.704711 13375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:25.710222 13375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: Bootstrap starting.
I20260925 04:37:25.710405 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13307
I20260925 04:37:25.710556 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:25.710840 27835 external_mini_cluster.cc:1428] Setting key 2c06714ba8f7d474c65cf371b21662c3
I20260925 04:37:25.712986 13375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:25.713901 13375 log.cc:824] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:25.716144 13375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: No bootstrap required, opened a new log
I20260925 04:37:25.720609 13375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } }
I20260925 04:37:25.720924 13375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:25.720973 13375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eed85d8dc9d4f8eae960b1d60a61e56, State: Initialized, Role: FOLLOWER
I20260925 04:37:25.721460 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.712202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.27.46.254:36711)
0925 04:37:25.712884 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.712906 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.712950 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:37:25.713830 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.713854 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.714395 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.714714 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:25.715215 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.716573 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.717447 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.718171 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.718361 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.719142 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:25.719149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:25.719166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:25.719755 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:25.719833 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:25.719860 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:25.719954 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.720052 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.720326 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20260925 04:37:25.721570 13375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } }
I20260925 04:37:25.721742 13375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:25.721827 13375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:25.721918 13375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:25.722982 13375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } }
I20260925 04:37:25.723347 13375 leader_election.cc:302] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eed85d8dc9d4f8eae960b1d60a61e56; no voters:
I20260925 04:37:25.723701 13375 leader_election.cc:288] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:25.723798 13381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:25.724076 13381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [term 1 LEADER]: Becoming Leader. State: Replica: 1eed85d8dc9d4f8eae960b1d60a61e56, State: Running, Role: LEADER
I20260925 04:37:25.724486 13381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } }
I20260925 04:37:25.725005 13375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:25.726171 13383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eed85d8dc9d4f8eae960b1d60a61e56. Latest consensus state: current_term: 1 leader_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } } }
I20260925 04:37:25.726315 13383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:25.726616 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eed85d8dc9d4f8eae960b1d60a61e56" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36711 } } }
I20260925 04:37:25.726751 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:25.726763 13390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:25.729863 13390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:25.735184 13390 catalog_manager.cc:1347] Generated new cluster ID: c3e3d032ae7f44ddae54380b82fdc9f7
I20260925 04:37:25.735234 13390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:25.743106 13390 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:25.743976 13390 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:25.769642 13390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: Generated new TSK 0
I20260925 04:37:25.770602 13390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:25.785072 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--builtin_ntp_servers=127.27.46.212:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:25.896224 13400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:25.896477 13400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:25.896515 13400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:25.900480 13400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:25.900527 13400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:25.900578 13400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:25.905412 13400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:25.906478 13400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:25.907817 13400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:25.917238 13406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.917253 13405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:25.917253 13408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:25.917727 13400 server_base.cc:1034] running on GCE node
I20260925 04:37:25.918124 13400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:25.918824 13400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:25.920011 13400 hybrid_clock.cc:648] HybridClock initialized: now 1790311045919997 us; error 27 us; skew 500 ppm
I20260925 04:37:25.921876 13400 webserver.cc:460] Webserver started at http://127.27.46.193:45717/ using document root <none> and password file <none>
I20260925 04:37:25.922446 13400 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:25.922559 13400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:25.922798 13400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:25.924839 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "b315661f1018415fb9e307550958e2b5"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "d398112ecdcefa4f2d315bda7b4cf89b"
server_key_iv: "cb0c1f8edb8173c08da93bde817f1ce6"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.925413 13400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "b315661f1018415fb9e307550958e2b5"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "d398112ecdcefa4f2d315bda7b4cf89b"
server_key_iv: "cb0c1f8edb8173c08da93bde817f1ce6"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.929220 13400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:25.931646 13414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.932884 13400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:25.933037 13400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "b315661f1018415fb9e307550958e2b5"
format_stamp: "Formatted at 2026-09-25 04:37:25 on dist-test-slave-r8r3"
server_key: "d398112ecdcefa4f2d315bda7b4cf89b"
server_key_iv: "cb0c1f8edb8173c08da93bde817f1ce6"
server_key_version: "encryptionkey@0"
I20260925 04:37:25.933161 13400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:25.944059 13400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:25.944792 13400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:25.944979 13400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:25.945608 13400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:25.946511 13400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:25.946584 13400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.946647 13400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:25.946698 13400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:25.955184 13400 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:34813
I20260925 04:37:25.955204 13528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:34813 every 8 connection(s)
I20260925 04:37:25.956022 13400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:25.961577 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13400
I20260925 04:37:25.961733 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:25.962031 27835 external_mini_cluster.cc:1428] Setting key f9b23b04e7e4d065071b71f05166d2b1
I20260925 04:37:25.964011 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--builtin_ntp_servers=127.27.46.212:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:25.966066 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.957864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:40121 (local address 127.27.46.254:36711)
0925 04:37:25.957998 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:25.958002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:25.959016 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 04:37:25.960454 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:25.960458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:25.960519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:25.960625 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:25.961634 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.962301 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.963669 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.964043 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.964168 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.965574 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:25.965579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:25.965582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:25.965597 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:25.965622 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:25.965635 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:25.965676 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:25.965874 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 04:37:25.965936 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:37:25.966907 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:25.958248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:36711 (local address 127.27.46.193:40121)
0925 04:37:25.958876 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:25.958915 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:25.960231 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:25.960801 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:25.960808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:25.960889 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:25.961446 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.961468 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.962466 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.962469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.963532 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:25.963542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:25.964336 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:25.964342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:25.964597 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:25.965225 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:25.965230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:25.965282 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:25.965365 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:25.965746 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:25.965753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:25.965757 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:37:25.965871 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:37:25.966134 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260925 04:37:25.968191 13529 heartbeater.cc:347] Connected to a master server at 127.27.46.254:36711
I20260925 04:37:25.968474 13529 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:25.969031 13529 heartbeater.cc:511] Master 127.27.46.254:36711 requested a full tablet report, sending...
I20260925 04:37:25.970662 13339 ts_manager.cc:194] Registered new tserver with Master: b315661f1018415fb9e307550958e2b5 (127.27.46.193:34813)
I20260925 04:37:25.971820 13339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.193:40121
W20260925 04:37:26.077665 13532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:26.077946 13532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:26.078016 13532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:26.082113 13532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:26.082191 13532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:26.082288 13532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:26.087072 13532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:26.088160 13532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:26.089568 13532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:26.099208 13537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:26.099184 13540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:26.099207 13538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:26.099583 13532 server_base.cc:1034] running on GCE node
I20260925 04:37:26.100003 13532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:26.100618 13532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:26.101807 13532 hybrid_clock.cc:648] HybridClock initialized: now 1790311046101793 us; error 25 us; skew 500 ppm
I20260925 04:37:26.103521 13532 webserver.cc:460] Webserver started at http://127.27.46.194:41147/ using document root <none> and password file <none>
I20260925 04:37:26.104099 13532 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:26.104190 13532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:26.104424 13532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:26.106496 13532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "87275612480446809261d9665c819bcc"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "b5b43c43205797bb3662e9825adfa9d9"
server_key_iv: "67fc745716d49226551d1ca41097a211"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.107098 13532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "87275612480446809261d9665c819bcc"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "b5b43c43205797bb3662e9825adfa9d9"
server_key_iv: "67fc745716d49226551d1ca41097a211"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.110671 13532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:37:26.113010 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.114354 13532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:26.114494 13532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "87275612480446809261d9665c819bcc"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "b5b43c43205797bb3662e9825adfa9d9"
server_key_iv: "67fc745716d49226551d1ca41097a211"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.114607 13532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:26.131552 13532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:26.132249 13532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:26.132452 13532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:26.133044 13532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:26.134021 13532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:26.134089 13532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.134163 13532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:26.134210 13532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.142848 13532 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:41277
I20260925 04:37:26.142880 13660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:41277 every 8 connection(s)
I20260925 04:37:26.143762 13532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:26.151149 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13532
I20260925 04:37:26.151309 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:26.151633 27835 external_mini_cluster.cc:1428] Setting key 9f9e16690a7dbd911c48c3a870f583f3
I20260925 04:37:26.152817 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.145616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:44475 (local address 127.27.46.254:36711)
0925 04:37:26.145759 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.145763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.146776 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 04:37:26.148188 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.148192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.148262 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.148402 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:26.149270 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.149875 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.150980 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.151186 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.151291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.152323 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:26.152326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:26.152328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:26.152340 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:26.152359 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:26.152366 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:26.152455 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.152606 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.152670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:37:26.153789 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--builtin_ntp_servers=127.27.46.212:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:26.153888 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.146021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:36711 (local address 127.27.46.194:44475)
0925 04:37:26.146628 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.146666 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.147975 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.148575 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.148582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.148665 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:26.149122 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.149134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.150005 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.150009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.150859 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.150868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.151332 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.151336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.151491 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.152060 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:26.152061 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:26.152097 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:26.152153 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:26.152457 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:26.152468 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:26.152476 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.152606 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.152867 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
I20260925 04:37:26.155002 13661 heartbeater.cc:347] Connected to a master server at 127.27.46.254:36711
I20260925 04:37:26.155241 13661 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:26.155824 13661 heartbeater.cc:511] Master 127.27.46.254:36711 requested a full tablet report, sending...
I20260925 04:37:26.156910 13339 ts_manager.cc:194] Registered new tserver with Master: 87275612480446809261d9665c819bcc (127.27.46.194:41277)
I20260925 04:37:26.157476 13339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.194:44475
W20260925 04:37:26.271112 13664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:26.271535 13664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:26.271634 13664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:26.275823 13664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:26.275872 13664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:26.275956 13664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:26.280704 13664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:26.281879 13664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:26.283257 13664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:26.294416 13672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:26.294440 13669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:26.294534 13670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:26.294752 13664 server_base.cc:1034] running on GCE node
I20260925 04:37:26.295279 13664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:26.295953 13664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:26.297143 13664 hybrid_clock.cc:648] HybridClock initialized: now 1790311046297126 us; error 41 us; skew 500 ppm
I20260925 04:37:26.299041 13664 webserver.cc:460] Webserver started at http://127.27.46.195:44671/ using document root <none> and password file <none>
I20260925 04:37:26.299587 13664 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:26.299646 13664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:26.299854 13664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:26.301955 13664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "22187319550e4821b1a3880eb223b2c5"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "c15e8657ad4f9d1a693da12db229e7d1"
server_key_iv: "52dbde9bdfc46d680383669490521fdf"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.302522 13664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "22187319550e4821b1a3880eb223b2c5"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "c15e8657ad4f9d1a693da12db229e7d1"
server_key_iv: "52dbde9bdfc46d680383669490521fdf"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.306413 13664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:26.308943 13678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.310214 13664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:26.310338 13664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "22187319550e4821b1a3880eb223b2c5"
format_stamp: "Formatted at 2026-09-25 04:37:26 on dist-test-slave-r8r3"
server_key: "c15e8657ad4f9d1a693da12db229e7d1"
server_key_iv: "52dbde9bdfc46d680383669490521fdf"
server_key_version: "encryptionkey@0"
I20260925 04:37:26.310497 13664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:26.330896 13664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:26.331499 13664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:26.331673 13664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:26.332206 13664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:26.333158 13664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:26.333225 13664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.333290 13664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:26.333338 13664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:26.343851 13664 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:39213
I20260925 04:37:26.343869 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:39213 every 8 connection(s)
I20260925 04:37:26.344785 13664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:26.351053 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13664
I20260925 04:37:26.351190 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:26.351524 27835 external_mini_cluster.cc:1428] Setting key eb74ac7d8765b73043178b079803cdfb
I20260925 04:37:26.355648 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.346611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:55467 (local address 127.27.46.254:36711)
0925 04:37:26.346769 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.346774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.347935 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 04:37:26.349310 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.349316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.349395 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.349576 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:26.350428 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.351299 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.352991 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.353428 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.353615 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.354889 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:26.354896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:26.354903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:26.354922 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:26.354952 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:26.354968 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:26.355063 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.355339 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.355416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:37:26.356369 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.347009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:36711 (local address 127.27.46.195:55467)
0925 04:37:26.347760 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.347812 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.349077 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.349722 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.349728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.349810 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:26.350279 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.350291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.351448 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.351452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.352711 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.352722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.353689 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.353692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.353827 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.354587 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:26.354589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:26.354629 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:26.354694 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:26.355166 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:26.355175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:26.355180 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.355303 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.355521 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":136,"threads_started":1}
I20260925 04:37:26.357530 13794 heartbeater.cc:347] Connected to a master server at 127.27.46.254:36711
I20260925 04:37:26.357755 13794 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:26.358269 13794 heartbeater.cc:511] Master 127.27.46.254:36711 requested a full tablet report, sending...
I20260925 04:37:26.359383 13339 ts_manager.cc:194] Registered new tserver with Master: 22187319550e4821b1a3880eb223b2c5 (127.27.46.195:39213)
I20260925 04:37:26.359963 13339 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.195:55467
I20260925 04:37:26.365727 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:26.372057 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.367674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.27.46.254:36711)
0925 04:37:26.367833 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.367841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.367978 (+ 137us) server_negotiation.cc:407] Connection header received
0925 04:37:26.368063 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.368068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.368130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.368248 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:26.368501 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.369369 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.370236 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.370661 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.370790 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.371577 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:26.371592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:26.371597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:26.371615 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:26.371641 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:26.371654 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:26.371738 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.371799 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.371864 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:37:26.375025 13339 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:26.380683 13339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:26.396489 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.390030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39213 (local address 127.0.0.1:38030)
0925 04:37:26.391130 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.391142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.391220 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.391830 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.391832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.391849 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.391990 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.391999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.393304 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.393307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.394652 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.394662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.395954 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.395957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.396062 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.396100 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.396211 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.396315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260925 04:37:26.397094 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.390432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.27.46.195:39213)
0925 04:37:26.391517 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.391527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.391544 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:37:26.391597 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.391600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.391711 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.391847 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.392110 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.393184 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.394790 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.395594 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.395696 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.395892 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.396833 (+ 941us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.396941 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":143,"threads_started":1}
I20260925 04:37:26.397897 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.389840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.27.46.194:41277)
0925 04:37:26.390332 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.390349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.390910 (+ 561us) server_negotiation.cc:407] Connection header received
0925 04:37:26.391210 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.391219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.391455 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.391579 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.392117 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.393457 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.395432 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.396453 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.396560 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.396756 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.397547 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.397680 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":118,"threads_started":1}
I20260925 04:37:26.397411 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.389637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:41277 (local address 127.0.0.1:59614)
0925 04:37:26.390745 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.390797 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.391020 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.391803 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.391809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.391836 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.391990 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.391999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.393645 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.393649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.395269 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.395282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.396861 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.396866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.396984 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.397016 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.397083 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.397208 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":68,"threads_started":1}
I20260925 04:37:26.398823 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.390686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.27.46.193:34813)
0925 04:37:26.392109 (+ 1423us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.392123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.392329 (+ 206us) server_negotiation.cc:407] Connection header received
0925 04:37:26.392515 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.392536 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.392709 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.392817 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.393205 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.395058 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.396404 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.397987 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.398061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.398218 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.398589 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.398681 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":116,"threads_started":1}
I20260925 04:37:26.399125 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.390319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34813 (local address 127.0.0.1:54580)
0925 04:37:26.392241 (+ 1922us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.392253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.392394 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.392978 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.392981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.392996 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.393071 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.393076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.395186 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.395189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.396281 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.396289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.398355 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.398358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.398446 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.398466 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.398952 (+ 486us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.399022 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1870,"thread_start_us":82,"threads_started":1}
I20260925 04:37:26.400637 13727 tablet_service.cc:1467] Processing CreateTablet for tablet 391def866e64427d8cd83bd593da5ab5 (DEFAULT_TABLE table=test-table [id=a046e93714e9496f9e459faa0ff7e09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:26.401202 13594 tablet_service.cc:1467] Processing CreateTablet for tablet 391def866e64427d8cd83bd593da5ab5 (DEFAULT_TABLE table=test-table [id=a046e93714e9496f9e459faa0ff7e09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:26.401777 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 391def866e64427d8cd83bd593da5ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:26.402155 13594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 391def866e64427d8cd83bd593da5ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:26.402252 13462 tablet_service.cc:1467] Processing CreateTablet for tablet 391def866e64427d8cd83bd593da5ab5 (DEFAULT_TABLE table=test-table [id=a046e93714e9496f9e459faa0ff7e09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:26.403192 13462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 391def866e64427d8cd83bd593da5ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:26.410079 13812 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Bootstrap starting.
I20260925 04:37:26.410883 13813 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Bootstrap starting.
I20260925 04:37:26.412070 13811 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Bootstrap starting.
I20260925 04:37:26.412657 13812 tablet_bootstrap.cc:654] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:26.412957 13813 tablet_bootstrap.cc:654] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:26.413602 13812 log.cc:824] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:26.414359 13813 log.cc:824] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:26.414652 13811 tablet_bootstrap.cc:654] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:26.415504 13811 log.cc:824] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:26.415705 13812 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: No bootstrap required, opened a new log
I20260925 04:37:26.415886 13812 ts_tablet_manager.cc:1397] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:37:26.416357 13813 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: No bootstrap required, opened a new log
I20260925 04:37:26.416525 13813 ts_tablet_manager.cc:1397] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:37:26.417769 13811 tablet_bootstrap.cc:492] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: No bootstrap required, opened a new log
I20260925 04:37:26.417943 13811 ts_tablet_manager.cc:1397] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:37:26.420138 13812 raft_consensus.cc:348] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.420138 13813 raft_consensus.cc:348] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.420374 13813 raft_consensus.cc:374] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:26.420374 13812 raft_consensus.cc:374] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:26.420423 13812 raft_consensus.cc:729] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22187319550e4821b1a3880eb223b2c5, State: Initialized, Role: FOLLOWER
I20260925 04:37:26.420430 13813 raft_consensus.cc:729] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b315661f1018415fb9e307550958e2b5, State: Initialized, Role: FOLLOWER
I20260925 04:37:26.420904 13813 consensus_queue.cc:260] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.420904 13812 consensus_queue.cc:260] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.421873 13812 ts_tablet_manager.cc:1428] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:37:26.421873 13813 ts_tablet_manager.cc:1428] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:37:26.422200 13794 heartbeater.cc:503] Master 127.27.46.254:36711 was elected leader, sending a full tablet report...
I20260925 04:37:26.422200 13529 heartbeater.cc:503] Master 127.27.46.254:36711 was elected leader, sending a full tablet report...
I20260925 04:37:26.424464 13811 raft_consensus.cc:348] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.424840 13811 raft_consensus.cc:374] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:26.424949 13811 raft_consensus.cc:729] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87275612480446809261d9665c819bcc, State: Initialized, Role: FOLLOWER
I20260925 04:37:26.425480 13811 consensus_queue.cc:260] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.426607 13811 ts_tablet_manager.cc:1428] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 04:37:26.426671 13661 heartbeater.cc:503] Master 127.27.46.254:36711 was elected leader, sending a full tablet report...
W20260925 04:37:26.457235 13530 tablet.cc:2369] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:37:26.596257 13795 tablet.cc:2369] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:37:26.644886 13662 tablet.cc:2369] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:26.693662 13818 raft_consensus.cc:482] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:26.693872 13818 raft_consensus.cc:504] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.695174 13818 leader_election.cc:288] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87275612480446809261d9665c819bcc (127.27.46.194:41277), b315661f1018415fb9e307550958e2b5 (127.27.46.193:34813)
I20260925 04:37:26.698076 13819 raft_consensus.cc:482] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:26.698410 13819 raft_consensus.cc:504] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.700546 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.695357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:41277 (local address 127.27.46.195:36031)
0925 04:37:26.695495 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.695511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.695634 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.696495 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.696501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.696524 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.696653 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.696661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.697795 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.697800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.699147 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.699157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.700063 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.700066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.700171 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.700233 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.700319 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.700400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":62}
I20260925 04:37:26.700630 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.695845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:37687 (local address 127.27.46.193:34813)
0925 04:37:26.695977 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.695983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.696220 (+ 237us) server_negotiation.cc:407] Connection header received
0925 04:37:26.696383 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.696386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.696473 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.696555 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.696979 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.697731 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.699291 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.699863 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.699951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.700001 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.700453 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.700521 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:37:26.700929 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.696101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:36031 (local address 127.27.46.194:41277)
0925 04:37:26.696252 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.696257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.696272 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:37:26.696332 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.696335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.696387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.696466 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.696786 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.697656 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.699293 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.699857 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.699962 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.700001 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.700784 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.700843 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:37:26.700924 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.695700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34813 (local address 127.27.46.195:37687)
0925 04:37:26.696119 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.696135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.696264 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.696589 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.696593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.696615 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.696834 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.696848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.697855 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.697858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.699147 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.699157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.700089 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.700092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.700207 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.700337 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.700727 (+ 390us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.700809 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":185,"thread_start_us":120,"threads_started":1}
I20260925 04:37:26.701486 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "22187319550e4821b1a3880eb223b2c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87275612480446809261d9665c819bcc" is_pre_election: true
I20260925 04:37:26.701786 13614 raft_consensus.cc:2454] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22187319550e4821b1a3880eb223b2c5 in term 0.
I20260925 04:37:26.702235 13681 leader_election.cc:302] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22187319550e4821b1a3880eb223b2c5, 87275612480446809261d9665c819bcc; no voters:
I20260925 04:37:26.702492 13818 raft_consensus.cc:2781] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:26.702615 13818 raft_consensus.cc:482] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:26.702682 13818 raft_consensus.cc:3037] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:26.703976 13818 raft_consensus.cc:504] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.704653 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "22187319550e4821b1a3880eb223b2c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87275612480446809261d9665c819bcc"
I20260925 04:37:26.704771 13614 raft_consensus.cc:3037] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:26.705991 13614 raft_consensus.cc:2454] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22187319550e4821b1a3880eb223b2c5 in term 1.
I20260925 04:37:26.706079 13483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "22187319550e4821b1a3880eb223b2c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b315661f1018415fb9e307550958e2b5" is_pre_election: true
I20260925 04:37:26.706310 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.699919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34813 (local address 127.27.46.194:58341)
0925 04:37:26.700116 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.700130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.700216 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.702393 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.702395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.702414 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.702506 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.702511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.703393 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.703396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.704807 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.704816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.705683 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.705689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.705799 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.705825 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.706098 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.706174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260925 04:37:26.706341 13681 leader_election.cc:302] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22187319550e4821b1a3880eb223b2c5, 87275612480446809261d9665c819bcc; no voters:
I20260925 04:37:26.706411 13483 raft_consensus.cc:2454] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22187319550e4821b1a3880eb223b2c5 in term 0.
I20260925 04:37:26.706611 13822 raft_consensus.cc:2781] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:26.706662 13817 raft_consensus.cc:482] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:26.706744 13817 raft_consensus.cc:504] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.706822 13822 raft_consensus.cc:686] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 1 LEADER]: Becoming Leader. State: Replica: 22187319550e4821b1a3880eb223b2c5, State: Running, Role: LEADER
I20260925 04:37:26.707006 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.701825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:58341 (local address 127.27.46.193:34813)
0925 04:37:26.702191 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.702194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.702206 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:37:26.702245 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.702250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.702304 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.702375 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.702626 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.703217 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.704940 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.705440 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.705547 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.705589 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.705947 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.706023 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260925 04:37:26.707119 13822 consensus_queue.cc:237] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } }
I20260925 04:37:26.707337 13483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "87275612480446809261d9665c819bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b315661f1018415fb9e307550958e2b5" is_pre_election: true
I20260925 04:37:26.707458 13483 raft_consensus.cc:2454] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87275612480446809261d9665c819bcc in term 0.
I20260925 04:37:26.708115 13549 leader_election.cc:302] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87275612480446809261d9665c819bcc, b315661f1018415fb9e307550958e2b5; no voters:
I20260925 04:37:26.708532 13825 raft_consensus.cc:2741] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 04:37:26.709134 13817 leader_election.cc:288] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87275612480446809261d9665c819bcc (127.27.46.194:41277), 22187319550e4821b1a3880eb223b2c5 (127.27.46.195:39213)
I20260925 04:37:26.709393 13818 leader_election.cc:288] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [CANDIDATE]: Term 1 election: Requested vote from peers 87275612480446809261d9665c819bcc (127.27.46.194:41277), b315661f1018415fb9e307550958e2b5 (127.27.46.193:34813)
I20260925 04:37:26.709607 13819 leader_election.cc:288] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b315661f1018415fb9e307550958e2b5 (127.27.46.193:34813), 22187319550e4821b1a3880eb223b2c5 (127.27.46.195:39213)
I20260925 04:37:26.710248 13483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "22187319550e4821b1a3880eb223b2c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b315661f1018415fb9e307550958e2b5"
I20260925 04:37:26.710357 13483 raft_consensus.cc:3037] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:26.710593 13339 catalog_manager.cc:5495] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22187319550e4821b1a3880eb223b2c5 (127.27.46.195). New cstate: current_term: 1 leader_uuid: "22187319550e4821b1a3880eb223b2c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22187319550e4821b1a3880eb223b2c5" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 39213 } health_report { overall_health: HEALTHY } } }
I20260925 04:37:26.711602 13483 raft_consensus.cc:2454] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22187319550e4821b1a3880eb223b2c5 in term 1.
I20260925 04:37:26.713567 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.708200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:54227 (local address 127.27.46.194:41277)
0925 04:37:26.708620 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.708624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.709894 (+ 1270us) server_negotiation.cc:407] Connection header received
0925 04:37:26.710065 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.710068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.710107 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.710149 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.710531 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.711134 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.712458 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.712886 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.712969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.713005 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.713406 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.713457 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260925 04:37:26.713583 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.707925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:41277 (local address 127.27.46.193:54227)
0925 04:37:26.709820 (+ 1895us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.709837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.709959 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.710288 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.710291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.710308 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.710406 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.710411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.711281 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.711284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.712271 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.712281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.713141 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.713147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.713245 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.713279 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.713390 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.713450 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1831,"thread_start_us":68,"threads_started":1}
I20260925 04:37:26.713964 13614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "b315661f1018415fb9e307550958e2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87275612480446809261d9665c819bcc" is_pre_election: true
I20260925 04:37:26.714121 13614 raft_consensus.cc:2379] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b315661f1018415fb9e307550958e2b5 in current term 1: Already voted for candidate 22187319550e4821b1a3880eb223b2c5 in this term.
I20260925 04:37:26.717587 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.709818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39213 (local address 127.27.46.194:57281)
0925 04:37:26.712050 (+ 2232us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.712063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.712129 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.714295 (+ 2166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.714297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.714313 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.714376 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.714382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.715471 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.715474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.716363 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.716372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.717192 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.717195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.717287 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.717312 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.717375 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.717434 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165}
I20260925 04:37:26.717602 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.710010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:57281 (local address 127.27.46.195:39213)
0925 04:37:26.714082 (+ 4072us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.714090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.714102 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:37:26.714138 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.714140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.714179 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.714657 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.714745 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.715375 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.716487 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.717025 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.717272 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.717346 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.717386 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.717441 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":4012,"thread_start_us":96,"threads_started":1}
I20260925 04:37:26.718173 13747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "87275612480446809261d9665c819bcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22187319550e4821b1a3880eb223b2c5" is_pre_election: true
I20260925 04:37:26.718719 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.708998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:34571 (local address 127.27.46.195:39213)
0925 04:37:26.709092 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.709096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.714399 (+ 5303us) server_negotiation.cc:407] Connection header received
0925 04:37:26.714441 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.714445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.714486 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.714548 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:26.715338 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.715979 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.717163 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.717809 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.717900 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.717927 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.718553 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.718614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":37}
I20260925 04:37:26.719089 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.708960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:39213 (local address 127.27.46.193:34571)
0925 04:37:26.713797 (+ 4837us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.713808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:26.713895 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:26.715153 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:26.715155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:26.715173 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:26.715234 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.715238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.716091 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.716093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.717017 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:26.717026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.718255 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.718262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:26.718422 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.718442 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:37:26.718901 (+ 459us) client_negotiation.cc:240] Negotiation successful
0925 04:37:26.718965 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4790,"thread_start_us":69,"threads_started":1}
I20260925 04:37:26.719514 13747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "391def866e64427d8cd83bd593da5ab5" candidate_uuid: "b315661f1018415fb9e307550958e2b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22187319550e4821b1a3880eb223b2c5" is_pre_election: true
I20260925 04:37:26.719897 13418 leader_election.cc:302] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b315661f1018415fb9e307550958e2b5; no voters: 22187319550e4821b1a3880eb223b2c5, 87275612480446809261d9665c819bcc
I20260925 04:37:26.720144 13817 raft_consensus.cc:2726] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:37:26.762189 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.758980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.27.46.195:39213)
0925 04:37:26.759126 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:26.759129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:26.759143 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:37:26.759216 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:26.759218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:26.759275 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:26.759361 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:26.759568 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.760062 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.760841 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:26.761036 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:26.761107 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:26.761411 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:26.761834 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:26.761919 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:26.761978 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 04:37:26.762018 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:37:26.768620 13614 raft_consensus.cc:1261] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Refusing update from remote peer 22187319550e4821b1a3880eb223b2c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:26.768661 13483 raft_consensus.cc:1261] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 1 FOLLOWER]: Refusing update from remote peer 22187319550e4821b1a3880eb223b2c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:26.769310 13822 consensus_queue.cc:1035] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b315661f1018415fb9e307550958e2b5" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:26.769546 13818 consensus_queue.cc:1035] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87275612480446809261d9665c819bcc" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:26.778278 13831 mvcc.cc:205] Tried to move back new op lower bound from 7333114047555723264 to 7333114047315320832. Current Snapshot: MvccSnapshot[applied={T|T < 7333114047555723264}]
I20260925 04:37:26.780915 13832 mvcc.cc:205] Tried to move back new op lower bound from 7333114047555723264 to 7333114047315320832. Current Snapshot: MvccSnapshot[applied={T|T < 7333114047555723264 or (T in {7333114047555723264})}]
I20260925 04:37:26.785813 13339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:26.785946 13339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:26.788304 13339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eed85d8dc9d4f8eae960b1d60a61e56: Sending DeleteTablet for 3 replicas of tablet 391def866e64427d8cd83bd593da5ab5
I20260925 04:37:26.789265 13462 tablet_service.cc:1514] Processing DeleteTablet for tablet 391def866e64427d8cd83bd593da5ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:26 UTC) from {username='slave'} at 127.0.0.1:54580
I20260925 04:37:26.789273 13594 tablet_service.cc:1514] Processing DeleteTablet for tablet 391def866e64427d8cd83bd593da5ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:26 UTC) from {username='slave'} at 127.0.0.1:59614
I20260925 04:37:26.789758 13839 tablet_replica.cc:330] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5: stopping tablet replica
I20260925 04:37:26.789798 13727 tablet_service.cc:1514] Processing DeleteTablet for tablet 391def866e64427d8cd83bd593da5ab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:26 UTC) from {username='slave'} at 127.0.0.1:38030
I20260925 04:37:26.790143 13839 raft_consensus.cc:2229] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:26.790280 13841 tablet_replica.cc:330] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: stopping tablet replica
I20260925 04:37:26.790467 13839 raft_consensus.cc:2258] T 391def866e64427d8cd83bd593da5ab5 P b315661f1018415fb9e307550958e2b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:26.790586 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13400
I20260925 04:37:26.790711 13841 raft_consensus.cc:2229] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:37:26.791277 13841 raft_consensus.cc:2258] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:26.792706 13841 ts_tablet_manager.cc:1905] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:26.795965 13840 tablet_replica.cc:330] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: stopping tablet replica
I20260925 04:37:26.795986 13841 ts_tablet_manager.cc:1918] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:26.796072 13841 log.cc:1196] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/391def866e64427d8cd83bd593da5ab5
I20260925 04:37:26.796295 13840 raft_consensus.cc:2229] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:26.796370 13841 ts_tablet_manager.cc:1939] T 391def866e64427d8cd83bd593da5ab5 P 22187319550e4821b1a3880eb223b2c5: Deleting consensus metadata
I20260925 04:37:26.796599 13840 raft_consensus.cc:2258] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:26.797554 13325 catalog_manager.cc:4841] TS 22187319550e4821b1a3880eb223b2c5 (127.27.46.195:39213): tablet 391def866e64427d8cd83bd593da5ab5 (table test-table [id=a046e93714e9496f9e459faa0ff7e09a]) successfully deleted
I20260925 04:37:26.798204 13840 ts_tablet_manager.cc:1905] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:37:26.801065 13324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:34813 (error 108)
I20260925 04:37:26.801257 13840 ts_tablet_manager.cc:1918] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:26.801362 13840 log.cc:1196] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/391def866e64427d8cd83bd593da5ab5
I20260925 04:37:26.801692 13840 ts_tablet_manager.cc:1939] T 391def866e64427d8cd83bd593da5ab5 P 87275612480446809261d9665c819bcc: Deleting consensus metadata
I20260925 04:37:26.801947 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13532
I20260925 04:37:26.803960 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.803636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:34813 (local address 127.0.0.1:54584)
0925 04:37:26.803796 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.803869 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:34813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 04:37:26.804136 13324 catalog_manager.cc:4584] TS b315661f1018415fb9e307550958e2b5 (127.27.46.193:34813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 391def866e64427d8cd83bd593da5ab5: Network error: Client connection negotiation failed: client connection to 127.27.46.193:34813: connect: Connection refused (error 111)
I20260925 04:37:26.810494 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13664
I20260925 04:37:26.811719 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:26.810612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:41277 (local address 127.0.0.1:59630)
0925 04:37:26.811595 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:26.811647 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905}
I20260925 04:37:26.816896 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13307
2026-09-25T04:37:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1321 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:26 dist-test-slave-r8r3 krb5kdc[13846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:26 dist-test-slave-r8r3 krb5kdc[13846](info): set up 2 sockets
Sep 25 04:37:26 dist-test-slave-r8r3 krb5kdc[13846](info): commencing operation
krb5kdc: starting...
W20260925 04:37:27.837091 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:27 dist-test-slave-r8r3 krb5kdc[13846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:27.957311 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33355
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:33355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:28.068327 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:28.068583 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:28.068619 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:28.072636 13859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:28.072686 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:28.072705 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:28.072721 13859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:28.072734 13859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:28.077554 13859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:33355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:28.078483 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:28.079418 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:28.087710 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.087723 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.087755 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:28.087881 13859 server_base.cc:1034] running on GCE node
I20260925 04:37:28.088524 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:28.089598 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:28.090754 13859 hybrid_clock.cc:648] HybridClock initialized: now 1790311048090735 us; error 28 us; skew 500 ppm
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:28.093327 13859 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:28.094247 13859 webserver.cc:460] Webserver started at http://127.27.46.254:38903/ using document root <none> and password file <none>
I20260925 04:37:28.094828 13859 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:28.094921 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:28.095152 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:28.097158 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "fea316de753ce6fec98097599965a6e4"
server_key_iv: "d4c1891221ac6652c915137c4d8d54ae"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.097756 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "fea316de753ce6fec98097599965a6e4"
server_key_iv: "d4c1891221ac6652c915137c4d8d54ae"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.101332 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:37:28.103741 13875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.104998 13859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:28.105157 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "fea316de753ce6fec98097599965a6e4"
server_key_iv: "d4c1891221ac6652c915137c4d8d54ae"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.105284 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:28.119037 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:28.122165 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:28.122354 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:28.129272 13859 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:33355
I20260925 04:37:28.129359 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:33355 every 8 connection(s)
I20260925 04:37:28.130246 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:28.132892 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:28.133658 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13859
I20260925 04:37:28.133795 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:28.134092 27835 external_mini_cluster.cc:1428] Setting key d4893cf45f16ccd4e3aabd73b34f8cce
I20260925 04:37:28.139539 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: Bootstrap starting.
I20260925 04:37:28.141986 13928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:28.142673 13928 log.cc:824] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:28.144443 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: No bootstrap required, opened a new log
I20260925 04:37:28.148032 13928 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } }
I20260925 04:37:28.148249 13928 raft_consensus.cc:374] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:28.148311 13928 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29bb9fb7e4c44058f7d058e7ae92e6b, State: Initialized, Role: FOLLOWER
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:28.148784 13928 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } }
I20260925 04:37:28.148959 13928 raft_consensus.cc:388] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:28.149049 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:28.149168 13928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:28.150421 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } }
I20260925 04:37:28.150805 13928 leader_election.cc:302] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d29bb9fb7e4c44058f7d058e7ae92e6b; no voters:
I20260925 04:37:28.151161 13928 leader_election.cc:288] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:28.151311 13935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:28.151547 13935 raft_consensus.cc:686] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [term 1 LEADER]: Becoming Leader. State: Replica: d29bb9fb7e4c44058f7d058e7ae92e6b, State: Running, Role: LEADER
I20260925 04:37:28.151886 13935 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } }
I20260925 04:37:28.152593 13928 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:28.153122 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29bb9fb7e4c44058f7d058e7ae92e6b. Latest consensus state: current_term: 1 leader_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } } }
I20260925 04:37:28.153254 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:28.153658 13937 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29bb9fb7e4c44058f7d058e7ae92e6b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33355 } } }
I20260925 04:37:28.153786 13937 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:28.154103 13940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:28.155686 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.135416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.27.46.254:33355)
0925 04:37:28.135869 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.135886 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.135929 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:37:28.136736 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.136759 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.137181 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.137590 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:28.138319 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.139754 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.140609 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.141272 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.141421 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.149517 (+ 8096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:28.149524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:28.149541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:28.149574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:28.151603 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.152141 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.152147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.152153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.152301 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.152665 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.152669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.152670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.153267 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:28.153528 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.154449 (+ 921us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.154744 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260925 04:37:28.157173 13940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:28.163054 13940 catalog_manager.cc:1347] Generated new cluster ID: b6a5211228344dafb69092378e8e77ed
I20260925 04:37:28.163120 13940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:28.176061 13940 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:28.177389 13940 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:28.183964 13940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: Generated new TSK 0
I20260925 04:37:28.184751 13940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:37:28.216609 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:28.338382 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:28.338712 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:28.338796 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:28.342949 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:28.343037 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:28.343135 13957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:28.348645 13957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:28.349891 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:28.350997 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:28.361235 13963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.361199 13962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.361199 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:28.361550 13957 server_base.cc:1034] running on GCE node
I20260925 04:37:28.361977 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:28.362555 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:28.363723 13957 hybrid_clock.cc:648] HybridClock initialized: now 1790311048363707 us; error 26 us; skew 500 ppm
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:28.366591 13957 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:37:28.367457 13957 webserver.cc:460] Webserver started at http://127.27.46.193:38333/ using document root <none> and password file <none>
I20260925 04:37:28.368029 13957 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:28.368121 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:28.368358 13957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:28.370488 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "e3a9a1ab2dd54a72bbbd048e026de035"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "ab72254b986a4f338f94a251f0f9a56a"
server_key_iv: "09266d2a1d7cd55d36d3f37ac7a4881a"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.371104 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "e3a9a1ab2dd54a72bbbd048e026de035"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "ab72254b986a4f338f94a251f0f9a56a"
server_key_iv: "09266d2a1d7cd55d36d3f37ac7a4881a"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.375155 13957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:28.377562 13972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.378804 13957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:28.378947 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "e3a9a1ab2dd54a72bbbd048e026de035"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "ab72254b986a4f338f94a251f0f9a56a"
server_key_iv: "09266d2a1d7cd55d36d3f37ac7a4881a"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.379062 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:28.395608 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:28.399346 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:28.399569 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:28.400218 13957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:28.401118 13957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:28.401194 13957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.401270 13957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:28.401315 13957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.410974 13957 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:46351
I20260925 04:37:28.411006 14087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:46351 every 8 connection(s)
I20260925 04:37:28.411922 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:28.414032 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 13957
I20260925 04:37:28.414171 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:28.414556 27835 external_mini_cluster.cc:1428] Setting key 81580f61b2406519a5be887bdad38f40
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
I20260925 04:37:28.426817 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.413764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:45423 (local address 127.27.46.254:33355)
0925 04:37:28.413905 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.413908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.415025 (+ 1117us) server_negotiation.cc:407] Connection header received
0925 04:37:28.416643 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.416647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.416699 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.416774 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:28.418158 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.418736 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.419867 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.420114 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.420240 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.422933 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:28.422937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:28.422941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:28.422972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:28.424566 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.425175 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.425179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.425181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.425274 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.425701 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.425705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.425707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.425898 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:28.426010 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.426515 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.426640 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:37:28.427608 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.414281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33355 (local address 127.27.46.193:45423)
0925 04:37:28.414859 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.414902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.416412 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.416940 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.416947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.417303 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:28.417991 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.418012 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.418858 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.418860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.419741 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.419750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.420283 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.420287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.420512 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.421206 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:28.421222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:28.422768 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:28.424706 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.424712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.424746 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.425070 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.425376 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.425379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.425381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.425602 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.426008 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:28.426015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:28.426269 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.426455 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.426788 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:37:28.428974 14088 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33355
I20260925 04:37:28.429220 14088 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:28.429775 14088 heartbeater.cc:511] Master 127.27.46.254:33355 requested a full tablet report, sending...
I20260925 04:37:28.431295 13892 ts_manager.cc:194] Registered new tserver with Master: e3a9a1ab2dd54a72bbbd048e026de035 (127.27.46.193:46351)
I20260925 04:37:28.432366 13892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:45423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:37:28.445199 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:28.558020 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:28.558254 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:28.558296 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:28.562134 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:28.562184 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:28.562244 14093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:28.567211 14093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:28.568233 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:28.569175 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:28.578644 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.578697 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:28.578815 14093 server_base.cc:1034] running on GCE node
W20260925 04:37:28.578644 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:28.579298 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:28.579999 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:28.581197 14093 hybrid_clock.cc:648] HybridClock initialized: now 1790311048581176 us; error 32 us; skew 500 ppm
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:28.584028 14093 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:37:28.584847 14093 webserver.cc:460] Webserver started at http://127.27.46.194:36487/ using document root <none> and password file <none>
I20260925 04:37:28.585373 14093 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:28.585443 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:28.585667 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:28.587643 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "ba3e4e04cafd44e799bfa4ffb191b18b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "8784219c4dae0659becd307c3b221649"
server_key_iv: "a6275efeb0531ad0accce673eaa858cf"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.588166 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "ba3e4e04cafd44e799bfa4ffb191b18b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "8784219c4dae0659becd307c3b221649"
server_key_iv: "a6275efeb0531ad0accce673eaa858cf"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.591874 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:37:28.594221 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.595432 14093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:28.595554 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "ba3e4e04cafd44e799bfa4ffb191b18b"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "8784219c4dae0659becd307c3b221649"
server_key_iv: "a6275efeb0531ad0accce673eaa858cf"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.595664 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:28.612340 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:28.615324 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:28.615523 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:28.616111 14093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:28.616917 14093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:28.616963 14093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.617029 14093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:28.617069 14093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.626192 14093 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:42819
I20260925 04:37:28.626230 14224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:42819 every 8 connection(s)
I20260925 04:37:28.627072 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:28.631654 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14093
I20260925 04:37:28.631803 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:28.632083 27835 external_mini_cluster.cc:1428] Setting key adae0bb667842c7394e71a5611083c63
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:28.642201 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.628898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:39337 (local address 127.27.46.254:33355)
0925 04:37:28.629040 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.629044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.629892 (+ 848us) server_negotiation.cc:407] Connection header received
0925 04:37:28.631225 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.631229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.631286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.631398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:28.632834 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.633407 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.635104 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.635445 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.635564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.638084 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:28.638088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:28.638093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:28.638123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:28.639891 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.640519 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.640523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.640525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.640624 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.640972 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.640978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.640981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.641233 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:28.641337 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.641904 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.642038 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:37:28.642971 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.629272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33355 (local address 127.27.46.194:39337)
0925 04:37:28.629742 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.629784 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.631013 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.631586 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.631593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.632040 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:28.632656 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.632686 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.633570 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.633573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.634964 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.634979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.635596 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.635598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.635734 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.636328 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:28.636345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:28.637909 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:28.640041 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.640050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.640066 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.640349 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.640723 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.640725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.640726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.640865 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.641335 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:28.641340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:28.641635 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.641793 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.642044 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":169,"threads_started":1}
I20260925 04:37:28.644255 14225 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33355
I20260925 04:37:28.644518 14225 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
I20260925 04:37:28.645018 14225 heartbeater.cc:511] Master 127.27.46.254:33355 requested a full tablet report, sending...
I20260925 04:37:28.646154 13892 ts_manager.cc:194] Registered new tserver with Master: ba3e4e04cafd44e799bfa4ffb191b18b (127.27.46.194:42819)
I20260925 04:37:28.646728 13892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:39337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:37:28.660688 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:41551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:28.776257 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:28.776489 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:28.776531 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:28.780552 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:28.780617 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:28.780678 14230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:28.786365 14230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:41551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:28.787451 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:28.788481 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:28.798259 14235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.798273 14236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:28.798322 14238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:28.798338 14230 server_base.cc:1034] running on GCE node
I20260925 04:37:28.798957 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:28.799557 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:28.800755 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790311048800718 us; error 90 us; skew 500 ppm
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:28.803587 14230 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:37:28.804471 14230 webserver.cc:460] Webserver started at http://127.27.46.195:46427/ using document root <none> and password file <none>
I20260925 04:37:28.804996 14230 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:28.805053 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:28.805215 14230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:28.807332 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "6432e4a4b9f74234967f95b7eb031c05"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "4f1270671fd0ec41e5c41c3793407e65"
server_key_iv: "8533621846e4199d9f50826fbdff5790"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.807852 14230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "6432e4a4b9f74234967f95b7eb031c05"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "4f1270671fd0ec41e5c41c3793407e65"
server_key_iv: "8533621846e4199d9f50826fbdff5790"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.811618 14230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:28.814060 14245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.815295 14230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:28.815405 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "6432e4a4b9f74234967f95b7eb031c05"
format_stamp: "Formatted at 2026-09-25 04:37:28 on dist-test-slave-r8r3"
server_key: "4f1270671fd0ec41e5c41c3793407e65"
server_key_iv: "8533621846e4199d9f50826fbdff5790"
server_key_version: "encryptionkey@0"
I20260925 04:37:28.815497 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:28.847754 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:28.851213 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:28.851377 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:28.851933 14230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:28.852790 14230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:28.852838 14230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.852878 14230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:28.852893 14230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:28.861904 14230 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:42195
I20260925 04:37:28.861934 14361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:42195 every 8 connection(s)
I20260925 04:37:28.862826 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:28.867329 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14230
I20260925 04:37:28.867473 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:28.867815 27835 external_mini_cluster.cc:1428] Setting key 65385a4d35fac66bcfee361db96a544f
Sep 25 04:37:28 dist-test-slave-r8r3 krb5kdc[13846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:28.876758 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.864657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:60937 (local address 127.27.46.254:33355)
0925 04:37:28.864811 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.864814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.865736 (+ 922us) server_negotiation.cc:407] Connection header received
0925 04:37:28.867144 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.867147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.867194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.867273 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:28.868797 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.869390 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.870810 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.871014 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.871096 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.873562 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:28.873565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:28.873566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:28.873589 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:28.874994 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.875481 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.875483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.875485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.875555 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.875845 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.875847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.875848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.876010 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:28.876083 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.876545 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.876637 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:37:28.877810 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.865061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:33355 (local address 127.27.46.195:60937)
0925 04:37:28.865586 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.865639 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.866891 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.867530 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.867538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.868025 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:28.868642 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.868662 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.869559 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.869563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.870677 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.870687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.871177 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.871181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.871339 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.871890 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:28.871907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:28.873374 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:28.875109 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.875113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.875125 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.875387 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.875633 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:28.875635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:28.875636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:28.875765 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:28.876095 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:28.876100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:28.876340 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.876505 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.876759 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260925 04:37:28.879027 14362 heartbeater.cc:347] Connected to a master server at 127.27.46.254:33355
I20260925 04:37:28.879269 14362 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:28.879773 14362 heartbeater.cc:511] Master 127.27.46.254:33355 requested a full tablet report, sending...
I20260925 04:37:28.881093 13891 ts_manager.cc:194] Registered new tserver with Master: 6432e4a4b9f74234967f95b7eb031c05 (127.27.46.195:42195)
I20260925 04:37:28.881945 13891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:60937
I20260925 04:37:28.882022 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:28.892197 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.884303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.27.46.254:33355)
0925 04:37:28.884445 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.884452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.884664 (+ 212us) server_negotiation.cc:407] Connection header received
0925 04:37:28.884810 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.884813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.884873 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.884953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:28.885844 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.886351 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.887186 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.887424 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.887533 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.889057 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:28.889061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:28.889063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:28.889089 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:28.890351 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.890868 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.890872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.890873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.890956 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:28.891291 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:28.891295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:28.891296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:28.891504 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:28.891666 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.891935 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.892030 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:37:28.895361 13891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:28.898357 13891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:28.913331 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:46351 (local address 127.0.0.1:52916)
0925 04:37:28.907897 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.907952 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.908098 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.908930 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.908935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.908960 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:28.909195 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.909206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910408 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.910412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.911831 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.911843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.912746 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.912750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.912860 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.912897 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.913032 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.913151 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":112,"spinlock_wait_cycles":1920,"thread_start_us":82,"threads_started":1}
I20260925 04:37:28.913607 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.27.46.194:42819)
0925 04:37:28.908469 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.908480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.908503 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:37:28.908581 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.908588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.908808 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.909327 (+ 519us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:28.909457 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910678 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.912179 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.912932 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.913184 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.913304 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.913344 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.913441 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":102,"threads_started":1}
I20260925 04:37:28.913681 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.27.46.193:46351)
0925 04:37:28.908381 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.908398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.908425 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:37:28.908508 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.908514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.908746 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.909198 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:28.909356 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910281 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.911978 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.912619 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.912746 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.912900 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.913408 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.913538 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":123,"threads_started":1}
I20260925 04:37:28.914222 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:42195 (local address 127.0.0.1:42342)
0925 04:37:28.908564 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.908577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.908666 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.909330 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.909332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.909351 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:28.909429 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.909434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910845 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.910850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.911984 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.911993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.913849 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.913853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.913935 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.913957 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.914037 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.914108 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":36,"thread_start_us":96,"threads_started":1}
I20260925 04:37:28.914288 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.27.46.195:42195)
0925 04:37:28.908424 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:28.908436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:28.908767 (+ 331us) server_negotiation.cc:407] Connection header received
0925 04:37:28.908886 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:28.908894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:28.909057 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:28.909196 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:28.909557 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910678 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.912102 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.912932 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.913062 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.913206 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:28.914076 (+ 870us) server_negotiation.cc:299] Negotiation successful
0925 04:37:28.914153 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":142,"threads_started":1}
I20260925 04:37:28.913729 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:28.907340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42819 (local address 127.0.0.1:47236)
0925 04:37:28.907897 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:28.907951 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:28.908080 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:28.908954 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:28.908957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:28.908975 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:28.909175 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.909184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.910827 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.910831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.912084 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:28.912093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:28.913059 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:28.913062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:28.913159 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:28.913179 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:37:28.913240 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 04:37:28.913292 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":54,"thread_start_us":110,"threads_started":1}
I20260925 04:37:28.916564 14294 tablet_service.cc:1467] Processing CreateTablet for tablet d97503e52588448eb13396a03c254c81 (DEFAULT_TABLE table=test-table [id=233b54fdd0844a0c97c518145af1a89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:28.916611 14021 tablet_service.cc:1467] Processing CreateTablet for tablet d97503e52588448eb13396a03c254c81 (DEFAULT_TABLE table=test-table [id=233b54fdd0844a0c97c518145af1a89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:28.916759 14156 tablet_service.cc:1467] Processing CreateTablet for tablet d97503e52588448eb13396a03c254c81 (DEFAULT_TABLE table=test-table [id=233b54fdd0844a0c97c518145af1a89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:28.917416 14294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97503e52588448eb13396a03c254c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:28.917608 14156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97503e52588448eb13396a03c254c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:28.917603 14021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d97503e52588448eb13396a03c254c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:28.924258 14379 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Bootstrap starting.
I20260925 04:37:28.925356 14380 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: Bootstrap starting.
I20260925 04:37:28.925822 14381 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Bootstrap starting.
I20260925 04:37:28.926613 14379 tablet_bootstrap.cc:654] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:28.927121 14380 tablet_bootstrap.cc:654] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:28.927912 14381 tablet_bootstrap.cc:654] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:28.927950 14379 log.cc:824] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:28.928224 14380 log.cc:824] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:28.929335 14381 log.cc:824] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:28.930213 14379 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: No bootstrap required, opened a new log
I20260925 04:37:28.930402 14379 ts_tablet_manager.cc:1397] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:37:28.930408 14380 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: No bootstrap required, opened a new log
I20260925 04:37:28.930555 14380 ts_tablet_manager.cc:1397] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:37:28.931339 14381 tablet_bootstrap.cc:492] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: No bootstrap required, opened a new log
I20260925 04:37:28.931492 14381 ts_tablet_manager.cc:1397] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 04:37:28.933163 14380 raft_consensus.cc:348] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.933364 14380 raft_consensus.cc:374] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:28.933394 14380 raft_consensus.cc:729] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a9a1ab2dd54a72bbbd048e026de035, State: Initialized, Role: FOLLOWER
I20260925 04:37:28.933830 14380 consensus_queue.cc:260] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.934451 14088 heartbeater.cc:503] Master 127.27.46.254:33355 was elected leader, sending a full tablet report...
I20260925 04:37:28.934319 14379 raft_consensus.cc:348] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.934551 14379 raft_consensus.cc:374] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:28.934612 14379 raft_consensus.cc:729] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6432e4a4b9f74234967f95b7eb031c05, State: Initialized, Role: FOLLOWER
I20260925 04:37:28.934811 14380 ts_tablet_manager.cc:1428] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:28.935012 14379 consensus_queue.cc:260] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.935105 14381 raft_consensus.cc:348] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.935339 14381 raft_consensus.cc:374] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:28.935398 14381 raft_consensus.cc:729] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3e4e04cafd44e799bfa4ffb191b18b, State: Initialized, Role: FOLLOWER
I20260925 04:37:28.935870 14362 heartbeater.cc:503] Master 127.27.46.254:33355 was elected leader, sending a full tablet report...
I20260925 04:37:28.935803 14381 consensus_queue.cc:260] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:28.936744 14381 ts_tablet_manager.cc:1428] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:37:28.936995 14379 ts_tablet_manager.cc:1428] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:37:28.937075 14225 heartbeater.cc:503] Master 127.27.46.254:33355 was elected leader, sending a full tablet report...
I20260925 04:37:29.011407 14385 raft_consensus.cc:482] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:29.011713 14385 raft_consensus.cc:504] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:29.013126 14385 leader_election.cc:288] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba3e4e04cafd44e799bfa4ffb191b18b (127.27.46.194:42819), 6432e4a4b9f74234967f95b7eb031c05 (127.27.46.195:42195)
I20260925 04:37:29.017379 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.013681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:48895 (local address 127.27.46.195:42195)
0925 04:37:29.013932 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.013936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.014076 (+ 140us) server_negotiation.cc:407] Connection header received
0925 04:37:29.014264 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.014267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.014323 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.014426 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:29.014765 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.015219 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016343 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.016785 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.016968 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.017151 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.017219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260925 04:37:29.017392 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.013665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:43335 (local address 127.27.46.194:42819)
0925 04:37:29.013813 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.013819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.014021 (+ 202us) server_negotiation.cc:407] Connection header received
0925 04:37:29.014245 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.014248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.014303 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.014362 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:29.014715 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.015214 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016370 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.016795 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016926 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.016975 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.017190 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.017266 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:37:29.017418 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.013420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:42195 (local address 127.27.46.193:48895)
0925 04:37:29.013942 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.013956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.014112 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.014454 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.014456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.014481 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:29.014622 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.014627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.015331 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.015335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.016220 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.016229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.016916 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.017015 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.017038 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.017109 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.017202 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":67,"spinlock_wait_cycles":34176,"thread_start_us":257,"threads_started":1}
I20260925 04:37:29.017424 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.013420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:42819 (local address 127.27.46.193:43335)
0925 04:37:29.013944 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.013956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.014116 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.014412 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.014417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.014434 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:29.014601 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.014607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.015324 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.015330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.016229 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.016237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.016929 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.016933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.017038 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.017065 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.017154 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.017224 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":93,"spinlock_wait_cycles":11136,"thread_start_us":127,"threads_started":1}
I20260925 04:37:29.018064 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97503e52588448eb13396a03c254c81" candidate_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6432e4a4b9f74234967f95b7eb031c05" is_pre_election: true
I20260925 04:37:29.018407 14316 raft_consensus.cc:2454] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a9a1ab2dd54a72bbbd048e026de035 in term 0.
I20260925 04:37:29.018509 14178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97503e52588448eb13396a03c254c81" candidate_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" is_pre_election: true
I20260925 04:37:29.018779 14178 raft_consensus.cc:2454] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a9a1ab2dd54a72bbbd048e026de035 in term 0.
I20260925 04:37:29.019270 13973 leader_election.cc:302] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6432e4a4b9f74234967f95b7eb031c05, ba3e4e04cafd44e799bfa4ffb191b18b, e3a9a1ab2dd54a72bbbd048e026de035; no voters:
I20260925 04:37:29.019563 14385 raft_consensus.cc:2781] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:29.019660 14385 raft_consensus.cc:482] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:29.019692 14385 raft_consensus.cc:3037] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:29.020754 14385 raft_consensus.cc:504] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:29.021118 14385 leader_election.cc:288] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [CANDIDATE]: Term 1 election: Requested vote from peers ba3e4e04cafd44e799bfa4ffb191b18b (127.27.46.194:42819), 6432e4a4b9f74234967f95b7eb031c05 (127.27.46.195:42195)
I20260925 04:37:29.021464 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97503e52588448eb13396a03c254c81" candidate_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6432e4a4b9f74234967f95b7eb031c05"
I20260925 04:37:29.021457 14178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97503e52588448eb13396a03c254c81" candidate_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b"
I20260925 04:37:29.021654 14178 raft_consensus.cc:3037] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:29.021656 14316 raft_consensus.cc:3037] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:29.022735 14316 raft_consensus.cc:2454] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a9a1ab2dd54a72bbbd048e026de035 in term 1.
I20260925 04:37:29.022754 14178 raft_consensus.cc:2454] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a9a1ab2dd54a72bbbd048e026de035 in term 1.
I20260925 04:37:29.023057 13974 leader_election.cc:302] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6432e4a4b9f74234967f95b7eb031c05, e3a9a1ab2dd54a72bbbd048e026de035; no voters:
I20260925 04:37:29.023252 14385 raft_consensus.cc:2781] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:29.023512 14385 raft_consensus.cc:686] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 1 LEADER]: Becoming Leader. State: Replica: e3a9a1ab2dd54a72bbbd048e026de035, State: Running, Role: LEADER
I20260925 04:37:29.023847 14385 consensus_queue.cc:237] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } }
I20260925 04:37:29.027019 13891 catalog_manager.cc:5495] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a9a1ab2dd54a72bbbd048e026de035 (127.27.46.193). New cstate: current_term: 1 leader_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a9a1ab2dd54a72bbbd048e026de035" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 46351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 } health_report { overall_health: UNKNOWN } } }
I20260925 04:37:29.036640 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.033790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.27.46.193:46351)
0925 04:37:29.033916 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.033919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.033930 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:37:29.034061 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.034063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.034110 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.034188 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:29.034387 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.034823 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.035483 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.035651 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.035712 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.035935 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:29.036266 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:29.036358 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.036443 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.036508 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:37:29.042014 14178 raft_consensus.cc:1261] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 1 FOLLOWER]: Refusing update from remote peer e3a9a1ab2dd54a72bbbd048e026de035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:29.042022 14316 raft_consensus.cc:1261] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 1 FOLLOWER]: Refusing update from remote peer e3a9a1ab2dd54a72bbbd048e026de035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:29.042670 14391 consensus_queue.cc:1035] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6432e4a4b9f74234967f95b7eb031c05" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 42195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:29.042876 14385 consensus_queue.cc:1035] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba3e4e04cafd44e799bfa4ffb191b18b" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:29.059917 13891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:29.060087 13891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:29.062553 13891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d29bb9fb7e4c44058f7d058e7ae92e6b: Sending DeleteTablet for 3 replicas of tablet d97503e52588448eb13396a03c254c81
I20260925 04:37:29.063361 14156 tablet_service.cc:1514] Processing DeleteTablet for tablet d97503e52588448eb13396a03c254c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:47236
I20260925 04:37:29.063357 14021 tablet_service.cc:1514] Processing DeleteTablet for tablet d97503e52588448eb13396a03c254c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:52916
I20260925 04:37:29.063558 14294 tablet_service.cc:1514] Processing DeleteTablet for tablet d97503e52588448eb13396a03c254c81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:29 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:42342
I20260925 04:37:29.063872 14404 tablet_replica.cc:330] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035: stopping tablet replica
I20260925 04:37:29.064085 14405 tablet_replica.cc:330] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: stopping tablet replica
I20260925 04:37:29.064198 14404 raft_consensus.cc:2229] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:37:29.064378 14405 raft_consensus.cc:2229] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:29.064590 14404 raft_consensus.cc:2258] T d97503e52588448eb13396a03c254c81 P e3a9a1ab2dd54a72bbbd048e026de035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:29.064692 14405 raft_consensus.cc:2258] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:29.064970 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13957
I20260925 04:37:29.065073 14406 tablet_replica.cc:330] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: stopping tablet replica
I20260925 04:37:29.065356 14406 raft_consensus.cc:2229] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:29.065662 14406 raft_consensus.cc:2258] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:29.068177 14405 ts_tablet_manager.cc:1905] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:29.068190 14406 ts_tablet_manager.cc:1905] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:29.070590 14405 ts_tablet_manager.cc:1918] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:29.070734 14405 log.cc:1196] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/d97503e52588448eb13396a03c254c81
I20260925 04:37:29.071060 14405 ts_tablet_manager.cc:1939] T d97503e52588448eb13396a03c254c81 P ba3e4e04cafd44e799bfa4ffb191b18b: Deleting consensus metadata
I20260925 04:37:29.071604 14406 ts_tablet_manager.cc:1918] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:29.071720 14406 log.cc:1196] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/d97503e52588448eb13396a03c254c81
I20260925 04:37:29.072034 14406 ts_tablet_manager.cc:1939] T d97503e52588448eb13396a03c254c81 P 6432e4a4b9f74234967f95b7eb031c05: Deleting consensus metadata
I20260925 04:37:29.072355 13876 catalog_manager.cc:4841] TS ba3e4e04cafd44e799bfa4ffb191b18b (127.27.46.194:42819): tablet d97503e52588448eb13396a03c254c81 (table test-table [id=233b54fdd0844a0c97c518145af1a89d]) successfully deleted
I20260925 04:37:29.073417 13877 catalog_manager.cc:4841] TS 6432e4a4b9f74234967f95b7eb031c05 (127.27.46.195:42195): tablet d97503e52588448eb13396a03c254c81 (table test-table [id=233b54fdd0844a0c97c518145af1a89d]) successfully deleted
W20260925 04:37:29.073936 13879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:46351 (error 108)
I20260925 04:37:29.074047 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14093
I20260925 04:37:29.076261 14376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.075971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:46351 (local address 127.0.0.1:52928)
0925 04:37:29.076116 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.076184 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:46351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 04:37:29.076453 13879 catalog_manager.cc:4584] TS e3a9a1ab2dd54a72bbbd048e026de035 (127.27.46.193:46351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d97503e52588448eb13396a03c254c81: Network error: Client connection negotiation failed: client connection to 127.27.46.193:46351: connect: Connection refused (error 111)
I20260925 04:37:29.081015 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14230
I20260925 04:37:29.086918 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 13859
2026-09-25T04:37:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2271 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T04:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:29Z Disabled control of system clock
I20260925 04:37:29.117436 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45231
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:29.228616 14411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:29.228899 14411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:29.228936 14411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:29.233242 14411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:29.233309 14411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:29.233323 14411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:29.233340 14411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:29.233356 14411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:29.238211 14411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:45231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:45231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:29.239351 14411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:29.240782 14411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:29.249706 14416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.249699 14417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:29.249822 14411 server_base.cc:1034] running on GCE node
W20260925 04:37:29.249704 14419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:29.250634 14411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:29.251734 14411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:29.252925 14411 hybrid_clock.cc:648] HybridClock initialized: now 1790311049252912 us; error 35 us; skew 500 ppm
I20260925 04:37:29.254732 14411 webserver.cc:460] Webserver started at http://127.27.46.254:39297/ using document root <none> and password file <none>
I20260925 04:37:29.255312 14411 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:29.255374 14411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:29.255607 14411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:29.257712 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "885dd19d96be4ee78214074d1dafa957"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "0fc8d20a05896e86e6235eab9cf11204"
server_key_iv: "e8640a347f9aab045663bc6c75f20bec"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.258283 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "885dd19d96be4ee78214074d1dafa957"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "0fc8d20a05896e86e6235eab9cf11204"
server_key_iv: "e8640a347f9aab045663bc6c75f20bec"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.262063 14411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:37:29.264446 14426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.265744 14411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:29.265890 14411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "885dd19d96be4ee78214074d1dafa957"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "0fc8d20a05896e86e6235eab9cf11204"
server_key_iv: "e8640a347f9aab045663bc6c75f20bec"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.266001 14411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:29.283005 14411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:29.283733 14411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:29.283954 14411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:29.290807 14411 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:45231
I20260925 04:37:29.290805 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:45231 every 8 connection(s)
I20260925 04:37:29.291743 14411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:29.293179 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14411
I20260925 04:37:29.293305 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:29.293676 27835 external_mini_cluster.cc:1428] Setting key 25e2f8202fa344accc097481b6db382e
I20260925 04:37:29.295418 14481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:29.301062 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: Bootstrap starting.
I20260925 04:37:29.302980 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.295078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.27.46.254:45231)
0925 04:37:29.295781 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.295797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.295834 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:37:29.296483 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.296499 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.296885 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.297219 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:29.297718 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.298676 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.299579 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.300051 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.300226 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.301176 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:29.301184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:29.301201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:29.301663 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:29.301723 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:29.301744 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:29.301851 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.302019 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.302281 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":148,"threads_started":1}
I20260925 04:37:29.303498 14481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:29.304354 14481 log.cc:824] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:29.306368 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: No bootstrap required, opened a new log
I20260925 04:37:29.310225 14481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } }
I20260925 04:37:29.310431 14481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:29.310465 14481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885dd19d96be4ee78214074d1dafa957, State: Initialized, Role: FOLLOWER
I20260925 04:37:29.310869 14481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } }
I20260925 04:37:29.311010 14481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:29.311055 14481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:29.311125 14481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:29.312014 14481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } }
I20260925 04:37:29.312309 14481 leader_election.cc:302] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 885dd19d96be4ee78214074d1dafa957; no voters:
I20260925 04:37:29.312548 14481 leader_election.cc:288] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:29.312709 14486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:29.312960 14486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [term 1 LEADER]: Becoming Leader. State: Replica: 885dd19d96be4ee78214074d1dafa957, State: Running, Role: LEADER
I20260925 04:37:29.313298 14486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } }
I20260925 04:37:29.313802 14481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:29.315037 14487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "885dd19d96be4ee78214074d1dafa957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } } }
I20260925 04:37:29.315168 14487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:29.315033 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 885dd19d96be4ee78214074d1dafa957. Latest consensus state: current_term: 1 leader_uuid: "885dd19d96be4ee78214074d1dafa957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "885dd19d96be4ee78214074d1dafa957" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 45231 } } }
I20260925 04:37:29.315337 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:29.315660 14495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:29.318935 14495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:29.324677 14495 catalog_manager.cc:1347] Generated new cluster ID: e76ddc5154884e339928d45f07c05c0f
I20260925 04:37:29.324743 14495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:29.330516 14495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:29.331472 14495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:29.339866 14495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: Generated new TSK 0
I20260925 04:37:29.340559 14495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:29.347779 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--builtin_ntp_servers=127.27.46.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:29.461149 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:29.461397 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:29.461434 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:29.465691 14505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:29.465749 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:29.465821 14505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:29.470719 14505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:29.471844 14505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:29.473290 14505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:37:29.483024 14505 server_base.cc:1034] running on GCE node
W20260925 04:37:29.483006 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.483017 14510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.483043 14511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:29.483822 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:29.484488 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:29.485688 14505 hybrid_clock.cc:648] HybridClock initialized: now 1790311049485672 us; error 35 us; skew 500 ppm
I20260925 04:37:29.487686 14505 webserver.cc:460] Webserver started at http://127.27.46.193:42933/ using document root <none> and password file <none>
I20260925 04:37:29.488289 14505 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:29.488392 14505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:29.488646 14505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:29.490830 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "262dd6c0eda44da0acc667ed8b64fb22"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c78569d485d84ddd734ae740349d6563"
server_key_iv: "347655d97b13c8f145f80d477b192001"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.491443 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "262dd6c0eda44da0acc667ed8b64fb22"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c78569d485d84ddd734ae740349d6563"
server_key_iv: "347655d97b13c8f145f80d477b192001"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.495280 14505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:29.497815 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.499020 14505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:37:29.499190 14505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "262dd6c0eda44da0acc667ed8b64fb22"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c78569d485d84ddd734ae740349d6563"
server_key_iv: "347655d97b13c8f145f80d477b192001"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.499315 14505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:29.528806 14505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:29.529624 14505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:29.529848 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:29.530496 14505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:29.531515 14505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:29.531592 14505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.531672 14505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:29.531716 14505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.541235 14505 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:45497
I20260925 04:37:29.541262 14633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:45497 every 8 connection(s)
I20260925 04:37:29.542289 14505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:29.544549 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14505
I20260925 04:37:29.544694 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:29.544991 27835 external_mini_cluster.cc:1428] Setting key edaf43feaff267f75960cd6a1eb74f49
I20260925 04:37:29.547402 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--builtin_ntp_servers=127.27.46.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:29.553215 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.544256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:40647 (local address 127.27.46.254:45231)
0925 04:37:29.544463 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.544470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.546338 (+ 1868us) server_negotiation.cc:407] Connection header received
0925 04:37:29.547647 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.547652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.547717 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.547788 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:29.549221 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.549841 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.551136 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.551379 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.551494 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.552685 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:29.552689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:29.552691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:29.552710 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:29.552728 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:29.552736 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:29.552816 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.552988 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.553065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 04:37:29.554198 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.544812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45231 (local address 127.27.46.193:40647)
0925 04:37:29.545701 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.545755 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.547356 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.548224 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.548236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.548352 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:29.549079 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.549094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.549974 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.549977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.550990 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.550999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.551542 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.551546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.551724 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.552412 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:29.552413 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:29.552457 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:29.552523 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:29.552818 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:29.552831 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:29.552838 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.552976 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.553271 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":180,"threads_started":1}
I20260925 04:37:29.555524 14634 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45231
I20260925 04:37:29.555781 14634 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:29.556320 14634 heartbeater.cc:511] Master 127.27.46.254:45231 requested a full tablet report, sending...
I20260925 04:37:29.558106 14443 ts_manager.cc:194] Registered new tserver with Master: 262dd6c0eda44da0acc667ed8b64fb22 (127.27.46.193:45497)
I20260925 04:37:29.559857 14443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.193:40647
W20260925 04:37:29.663318 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:29.663648 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:29.663688 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:29.667644 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:29.667702 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:29.667758 14637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:29.672559 14637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:29.673738 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:29.675187 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:29.685019 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.685019 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.685041 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:29.685331 14637 server_base.cc:1034] running on GCE node
I20260925 04:37:29.685914 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:29.686585 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:29.689898 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790311049689873 us; error 33 us; skew 500 ppm
I20260925 04:37:29.692041 14637 webserver.cc:460] Webserver started at http://127.27.46.194:35331/ using document root <none> and password file <none>
I20260925 04:37:29.692683 14637 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:29.692785 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:29.693028 14637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:29.695256 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "43178a29d7bd4a6580dc0475dc5b2de7"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c36bea8c9e9a7006736195b4a575c037"
server_key_iv: "eac0d7474f5adde912174394578f0285"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.695863 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "43178a29d7bd4a6580dc0475dc5b2de7"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c36bea8c9e9a7006736195b4a575c037"
server_key_iv: "eac0d7474f5adde912174394578f0285"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.699734 14637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:29.702320 14651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.703648 14637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:29.703816 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "43178a29d7bd4a6580dc0475dc5b2de7"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c36bea8c9e9a7006736195b4a575c037"
server_key_iv: "eac0d7474f5adde912174394578f0285"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.703956 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:29.736392 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:29.737143 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:29.737365 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:29.738059 14637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:29.739041 14637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:29.739125 14637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.739195 14637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:29.739248 14637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.748100 14637 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:45367
I20260925 04:37:29.748121 14767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:45367 every 8 connection(s)
I20260925 04:37:29.749024 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:29.755220 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14637
I20260925 04:37:29.755388 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:29.755728 27835 external_mini_cluster.cc:1428] Setting key e941c0a6b4b05a2c594bbf9e8f5fea1d
I20260925 04:37:29.757851 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--builtin_ntp_servers=127.27.46.212:44801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:29.758682 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.750842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:53631 (local address 127.27.46.254:45231)
0925 04:37:29.751025 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.751030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.752158 (+ 1128us) server_negotiation.cc:407] Connection header received
0925 04:37:29.753549 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.753554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.753615 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.753690 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:29.754569 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.755179 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.756584 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.756820 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.756933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.758157 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:29.758161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:29.758164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:29.758177 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:29.758204 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:29.758213 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:29.758293 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.758459 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.758525 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:37:29.759799 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.751261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45231 (local address 127.27.46.194:53631)
0925 04:37:29.751998 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.752042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.753311 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.753859 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.753865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.753948 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:29.754435 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.754447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.755353 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.755358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.756452 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.756462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.756975 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.756979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.757139 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.757816 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:29.757817 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:29.757856 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:29.757942 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:29.758296 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:29.758308 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:29.758314 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.758445 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.758714 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":127,"threads_started":1}
I20260925 04:37:29.761003 14768 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45231
I20260925 04:37:29.761256 14768 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:29.761837 14768 heartbeater.cc:511] Master 127.27.46.254:45231 requested a full tablet report, sending...
I20260925 04:37:29.762893 14443 ts_manager.cc:194] Registered new tserver with Master: 43178a29d7bd4a6580dc0475dc5b2de7 (127.27.46.194:45367)
I20260925 04:37:29.763489 14443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.194:53631
W20260925 04:37:29.875257 14771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:29.875547 14771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:29.875614 14771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:29.879623 14771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:29.879719 14771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:29.879817 14771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:29.884752 14771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:44801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:45231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:29.885896 14771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:29.887359 14771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:29.897040 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.897017 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:29.897004 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:29.897261 14771 server_base.cc:1034] running on GCE node
I20260925 04:37:29.897814 14771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:29.898398 14771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:29.899554 14771 hybrid_clock.cc:648] HybridClock initialized: now 1790311049899522 us; error 43 us; skew 500 ppm
I20260925 04:37:29.901461 14771 webserver.cc:460] Webserver started at http://127.27.46.195:46047/ using document root <none> and password file <none>
I20260925 04:37:29.902091 14771 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:29.902153 14771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:29.902426 14771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:29.904739 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "374a18bf281048f6a2b9c6c843ebe43a"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c18fd478d0e47d931e20b527256727ae"
server_key_iv: "10709c29d5f226db858e6c7a0759e384"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.905318 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "374a18bf281048f6a2b9c6c843ebe43a"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c18fd478d0e47d931e20b527256727ae"
server_key_iv: "10709c29d5f226db858e6c7a0759e384"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.909263 14771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 04:37:29.911871 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.913141 14771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 04:37:29.913285 14771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "374a18bf281048f6a2b9c6c843ebe43a"
format_stamp: "Formatted at 2026-09-25 04:37:29 on dist-test-slave-r8r3"
server_key: "c18fd478d0e47d931e20b527256727ae"
server_key_iv: "10709c29d5f226db858e6c7a0759e384"
server_key_version: "encryptionkey@0"
I20260925 04:37:29.913410 14771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:29.931394 14771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:29.932148 14771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:29.932353 14771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:29.932981 14771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:29.934031 14771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:29.934101 14771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.934175 14771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:29.934221 14771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:29.943455 14771 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:46769
I20260925 04:37:29.943483 14900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:46769 every 8 connection(s)
I20260925 04:37:29.944393 14771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:29.945013 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14771
I20260925 04:37:29.945134 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:29.945441 27835 external_mini_cluster.cc:1428] Setting key eba5fe52face57b9340a9f0d0f4d0d84
I20260925 04:37:29.953857 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.946435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:35689 (local address 127.27.46.254:45231)
0925 04:37:29.946569 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.946574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.947809 (+ 1235us) server_negotiation.cc:407] Connection header received
0925 04:37:29.949185 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.949187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.949232 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.949297 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:29.950282 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.950804 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.952012 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.952222 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.952301 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.953403 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:29.953405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:29.953407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:29.953418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:29.953432 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:29.953439 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:29.953542 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.953690 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.953742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:37:29.954748 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.946938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:45231 (local address 127.27.46.195:35689)
0925 04:37:29.947670 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.947722 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.949008 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.949464 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.949470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.949629 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:29.950130 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.950142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.950919 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.950921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.951907 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.951917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.952363 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.952366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.952509 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.953156 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:29.953158 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:29.953199 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:29.953268 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:29.953557 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:29.953564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:29.953568 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.953668 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.953890 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260925 04:37:29.955904 14901 heartbeater.cc:347] Connected to a master server at 127.27.46.254:45231
I20260925 04:37:29.956111 14901 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:29.956593 14901 heartbeater.cc:511] Master 127.27.46.254:45231 requested a full tablet report, sending...
I20260925 04:37:29.957654 14443 ts_manager.cc:194] Registered new tserver with Master: 374a18bf281048f6a2b9c6c843ebe43a (127.27.46.195:46769)
I20260925 04:37:29.958225 14443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.195:35689
I20260925 04:37:29.959549 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:29.964860 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.961525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.27.46.254:45231)
0925 04:37:29.961629 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.961632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.961755 (+ 123us) server_negotiation.cc:407] Connection header received
0925 04:37:29.961858 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.961860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.961896 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.961960 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:29.962185 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.962662 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.963446 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.963642 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.963714 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.964475 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:29.964479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:29.964480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:29.964492 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:29.964509 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:29.964517 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:29.964618 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.964729 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.964758 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 04:37:29.967558 14443 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:29.970664 14443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:29.986063 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.979510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:46769 (local address 127.0.0.1:45404)
0925 04:37:29.980624 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.980675 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.980814 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.981475 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.981481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.981536 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:29.981734 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.981743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983197 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.983200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.984490 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.984500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.985428 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.985432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.985585 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.985637 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.985757 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.985898 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":230,"spinlock_wait_cycles":13184,"thread_start_us":107,"threads_started":1}
I20260925 04:37:29.986158 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.980239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.27.46.195:46769)
0925 04:37:29.981084 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.981092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.981110 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:37:29.981168 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.981171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.981317 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.981481 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:29.981891 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983075 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.984635 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.985293 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.985404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.985622 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.985834 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.985948 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":157,"threads_started":1}
I20260925 04:37:29.986441 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.979788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:45497 (local address 127.0.0.1:56646)
0925 04:37:29.980622 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.980675 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.980812 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.982220 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.982223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.982240 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:29.982323 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.982329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983596 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.983602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.984913 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.984924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.985900 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.985903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.986134 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.986166 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.986225 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.986291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":92,"spinlock_wait_cycles":16640,"thread_start_us":97,"threads_started":1}
I20260925 04:37:29.987267 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.979440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:45367 (local address 127.0.0.1:50444)
0925 04:37:29.980624 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:29.980675 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:29.980814 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:29.981474 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:29.981480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:29.981533 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:29.981722 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.981728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983197 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.983200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.984913 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:29.984924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.986851 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.986855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:29.986963 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.986992 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:37:29.987061 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:37:29.987129 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":179,"spinlock_wait_cycles":16512,"thread_start_us":76,"threads_started":1}
I20260925 04:37:29.987334 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.980220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.27.46.193:45497)
0925 04:37:29.981828 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.981843 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.981869 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:37:29.981940 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.981947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.982083 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.982242 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:29.982438 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983475 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.985068 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.985775 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.986800 (+ 1025us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.987003 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.987055 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.987150 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":92,"threads_started":1}
I20260925 04:37:29.988119 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:29.979553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.27.46.194:45367)
0925 04:37:29.981054 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:29.981081 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:29.981100 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:37:29.981160 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:29.981164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:29.981317 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:29.981481 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:29.981851 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.983055 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.985058 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:29.985966 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:29.986077 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:29.987824 (+ 1747us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:29.987876 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 04:37:29.987965 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":123,"threads_started":1}
I20260925 04:37:29.989277 14834 tablet_service.cc:1467] Processing CreateTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d (DEFAULT_TABLE table=test-table [id=60a995616e864895b3e2d9da23a1afd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:29.989862 14567 tablet_service.cc:1467] Processing CreateTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d (DEFAULT_TABLE table=test-table [id=60a995616e864895b3e2d9da23a1afd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:29.990481 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0534de2ecc074e1a9a971dbf6d10b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:29.990909 14567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0534de2ecc074e1a9a971dbf6d10b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:29.990864 14700 tablet_service.cc:1467] Processing CreateTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d (DEFAULT_TABLE table=test-table [id=60a995616e864895b3e2d9da23a1afd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:29.991806 14700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0534de2ecc074e1a9a971dbf6d10b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:29.997005 14918 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Bootstrap starting.
I20260925 04:37:29.998454 14919 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Bootstrap starting.
I20260925 04:37:29.998621 14920 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Bootstrap starting.
I20260925 04:37:30.000221 14918 tablet_bootstrap.cc:654] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:30.000432 14920 tablet_bootstrap.cc:654] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:30.000535 14919 tablet_bootstrap.cc:654] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:30.001163 14920 log.cc:824] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:30.001231 14919 log.cc:824] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:30.001127 14918 log.cc:824] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:30.003312 14919 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: No bootstrap required, opened a new log
I20260925 04:37:30.003373 14920 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: No bootstrap required, opened a new log
I20260925 04:37:30.003541 14919 ts_tablet_manager.cc:1397] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:37:30.003556 14920 ts_tablet_manager.cc:1397] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 04:37:30.003746 14918 tablet_bootstrap.cc:492] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: No bootstrap required, opened a new log
I20260925 04:37:30.003928 14918 ts_tablet_manager.cc:1397] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:37:30.007052 14919 raft_consensus.cc:348] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.007337 14919 raft_consensus.cc:374] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:30.007422 14919 raft_consensus.cc:729] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a18bf281048f6a2b9c6c843ebe43a, State: Initialized, Role: FOLLOWER
I20260925 04:37:30.007889 14919 consensus_queue.cc:260] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.007944 14920 raft_consensus.cc:348] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.008188 14920 raft_consensus.cc:374] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:30.008245 14920 raft_consensus.cc:729] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43178a29d7bd4a6580dc0475dc5b2de7, State: Initialized, Role: FOLLOWER
I20260925 04:37:30.008443 14918 raft_consensus.cc:348] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.008656 14920 consensus_queue.cc:260] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.008734 14918 raft_consensus.cc:374] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:30.008780 14918 raft_consensus.cc:729] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262dd6c0eda44da0acc667ed8b64fb22, State: Initialized, Role: FOLLOWER
I20260925 04:37:30.008960 14919 ts_tablet_manager.cc:1428] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:37:30.009298 14901 heartbeater.cc:503] Master 127.27.46.254:45231 was elected leader, sending a full tablet report...
I20260925 04:37:30.009338 14918 consensus_queue.cc:260] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.010017 14634 heartbeater.cc:503] Master 127.27.46.254:45231 was elected leader, sending a full tablet report...
I20260925 04:37:30.010321 14918 ts_tablet_manager.cc:1428] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 04:37:30.010669 14920 ts_tablet_manager.cc:1428] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260925 04:37:30.011014 14768 heartbeater.cc:503] Master 127.27.46.254:45231 was elected leader, sending a full tablet report...
W20260925 04:37:30.043648 14635 tablet.cc:2369] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:30.043682 14925 raft_consensus.cc:482] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:30.043880 14925 raft_consensus.cc:504] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.045111 14925 leader_election.cc:288] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 374a18bf281048f6a2b9c6c843ebe43a (127.27.46.195:46769), 262dd6c0eda44da0acc667ed8b64fb22 (127.27.46.193:45497)
I20260925 04:37:30.050086 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.045531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:35003 (local address 127.27.46.193:45497)
0925 04:37:30.045674 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:30.045679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:30.045962 (+ 283us) server_negotiation.cc:407] Connection header received
0925 04:37:30.046108 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:30.046113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:30.046166 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:30.046229 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:30.046526 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.047265 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.048717 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.049338 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.049485 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:30.049568 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:30.049872 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 04:37:30.049955 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:37:30.050086 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.045459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:41059 (local address 127.27.46.195:46769)
0925 04:37:30.045617 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:30.045624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:30.045639 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:37:30.045830 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:30.045833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:30.045878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:30.045978 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:30.046319 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.047093 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.048717 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.049341 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.049485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:30.049568 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:30.049872 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 04:37:30.049955 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:37:30.050087 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.045317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:46769 (local address 127.27.46.194:41059)
0925 04:37:30.045567 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:30.045583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:30.045694 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:30.046007 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:30.046012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:30.046031 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:30.046172 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:30.046177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.047214 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.047217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:30.048526 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:30.048538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.049485 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.049490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:30.049679 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:30.049724 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:37:30.049812 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:37:30.049905 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":72}
I20260925 04:37:30.050096 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.045417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:45497 (local address 127.27.46.194:35003)
0925 04:37:30.045863 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:30.045874 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:30.045977 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:30.046279 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:30.046282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:30.046303 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:30.046404 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:30.046409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.047388 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.047391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:30.048549 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:30.048555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.049514 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.049518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:30.049675 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:30.049723 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 04:37:30.049810 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:37:30.049905 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":74,"thread_start_us":150,"threads_started":1}
I20260925 04:37:30.050832 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0534de2ecc074e1a9a971dbf6d10b59d" candidate_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374a18bf281048f6a2b9c6c843ebe43a" is_pre_election: true
I20260925 04:37:30.050869 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0534de2ecc074e1a9a971dbf6d10b59d" candidate_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262dd6c0eda44da0acc667ed8b64fb22" is_pre_election: true
I20260925 04:37:30.051154 14854 raft_consensus.cc:2454] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43178a29d7bd4a6580dc0475dc5b2de7 in term 0.
I20260925 04:37:30.051192 14587 raft_consensus.cc:2454] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43178a29d7bd4a6580dc0475dc5b2de7 in term 0.
I20260925 04:37:30.051680 14655 leader_election.cc:302] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 262dd6c0eda44da0acc667ed8b64fb22, 43178a29d7bd4a6580dc0475dc5b2de7; no voters:
I20260925 04:37:30.052161 14925 raft_consensus.cc:2781] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:30.052228 14925 raft_consensus.cc:482] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:30.052287 14925 raft_consensus.cc:3037] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:30.053303 14925 raft_consensus.cc:504] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.053720 14925 leader_election.cc:288] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [CANDIDATE]: Term 1 election: Requested vote from peers 374a18bf281048f6a2b9c6c843ebe43a (127.27.46.195:46769), 262dd6c0eda44da0acc667ed8b64fb22 (127.27.46.193:45497)
I20260925 04:37:30.054028 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0534de2ecc074e1a9a971dbf6d10b59d" candidate_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374a18bf281048f6a2b9c6c843ebe43a"
I20260925 04:37:30.054090 14587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0534de2ecc074e1a9a971dbf6d10b59d" candidate_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "262dd6c0eda44da0acc667ed8b64fb22"
I20260925 04:37:30.054148 14854 raft_consensus.cc:3037] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:30.054272 14587 raft_consensus.cc:3037] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:30.055303 14854 raft_consensus.cc:2454] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43178a29d7bd4a6580dc0475dc5b2de7 in term 1.
I20260925 04:37:30.055531 14587 raft_consensus.cc:2454] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43178a29d7bd4a6580dc0475dc5b2de7 in term 1.
I20260925 04:37:30.055672 14652 leader_election.cc:302] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374a18bf281048f6a2b9c6c843ebe43a, 43178a29d7bd4a6580dc0475dc5b2de7; no voters:
I20260925 04:37:30.055863 14925 raft_consensus.cc:2781] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:30.056128 14925 raft_consensus.cc:686] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 1 LEADER]: Becoming Leader. State: Replica: 43178a29d7bd4a6580dc0475dc5b2de7, State: Running, Role: LEADER
I20260925 04:37:30.056509 14925 consensus_queue.cc:237] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } }
I20260925 04:37:30.059412 14442 catalog_manager.cc:5495] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43178a29d7bd4a6580dc0475dc5b2de7 (127.27.46.194). New cstate: current_term: 1 leader_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43178a29d7bd4a6580dc0475dc5b2de7" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 45367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 } health_report { overall_health: UNKNOWN } } }
I20260925 04:37:30.085050 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.081318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.27.46.194:45367)
0925 04:37:30.081478 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:30.081485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:30.081523 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:37:30.081607 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:30.081609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:30.081667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:30.081794 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:30.082080 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.082584 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.083495 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:30.083713 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:30.083840 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:30.084163 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:30.084574 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:30.084741 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:30.084843 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 04:37:30.084883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:37:30.090939 14854 raft_consensus.cc:1261] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 1 FOLLOWER]: Refusing update from remote peer 43178a29d7bd4a6580dc0475dc5b2de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:30.090939 14587 raft_consensus.cc:1261] T 0534de2ecc074e1a9a971dbf6d10b59d P 262dd6c0eda44da0acc667ed8b64fb22 [term 1 FOLLOWER]: Refusing update from remote peer 43178a29d7bd4a6580dc0475dc5b2de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:30.091681 14925 consensus_queue.cc:1035] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "374a18bf281048f6a2b9c6c843ebe43a" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 46769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:30.091902 14928 consensus_queue.cc:1035] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "262dd6c0eda44da0acc667ed8b64fb22" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 45497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:30.101940 14932 mvcc.cc:205] Tried to move back new op lower bound from 7333114061165109248 to 7333114061034340352. Current Snapshot: MvccSnapshot[applied={T|T < 7333114061165109248}]
I20260925 04:37:30.108716 14441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:30.108966 14441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:30.111539 14441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 885dd19d96be4ee78214074d1dafa957: Sending DeleteTablet for 3 replicas of tablet 0534de2ecc074e1a9a971dbf6d10b59d
I20260925 04:37:30.112385 14700 tablet_service.cc:1514] Processing DeleteTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:30 UTC) from {username='slave'} at 127.0.0.1:50444
I20260925 04:37:30.112466 14567 tablet_service.cc:1514] Processing DeleteTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:30 UTC) from {username='slave'} at 127.0.0.1:56646
I20260925 04:37:30.112489 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet 0534de2ecc074e1a9a971dbf6d10b59d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:30 UTC) from {username='slave'} at 127.0.0.1:45404
I20260925 04:37:30.112948 14940 tablet_replica.cc:330] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: stopping tablet replica
I20260925 04:37:30.113134 14942 tablet_replica.cc:330] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: stopping tablet replica
I20260925 04:37:30.113410 14940 raft_consensus.cc:2229] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:37:30.113612 14942 raft_consensus.cc:2229] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:30.113699 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14505
I20260925 04:37:30.114343 14942 raft_consensus.cc:2258] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:30.116019 14942 ts_tablet_manager.cc:1905] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:30.117686 14940 raft_consensus.cc:2258] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:30.118996 14942 ts_tablet_manager.cc:1918] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:30.119098 14942 log.cc:1196] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/0534de2ecc074e1a9a971dbf6d10b59d
I20260925 04:37:30.119417 14942 ts_tablet_manager.cc:1939] T 0534de2ecc074e1a9a971dbf6d10b59d P 374a18bf281048f6a2b9c6c843ebe43a: Deleting consensus metadata
I20260925 04:37:30.121042 14427 catalog_manager.cc:4841] TS 374a18bf281048f6a2b9c6c843ebe43a (127.27.46.195:46769): tablet 0534de2ecc074e1a9a971dbf6d10b59d (table test-table [id=60a995616e864895b3e2d9da23a1afd6]) successfully deleted
I20260925 04:37:30.121963 14940 ts_tablet_manager.cc:1905] T 0534de2ecc074e1a9a971dbf6d10b59d P 43178a29d7bd4a6580dc0475dc5b2de7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:37:30.123545 14430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:45497 (error 108)
I20260925 04:37:30.123802 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14637
I20260925 04:37:30.125758 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.125369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:45497 (local address 127.0.0.1:56660)
0925 04:37:30.125562 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:30.125646 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:45497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 04:37:30.126240 14430 catalog_manager.cc:4584] TS 262dd6c0eda44da0acc667ed8b64fb22 (127.27.46.193:45497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0534de2ecc074e1a9a971dbf6d10b59d: Network error: Client connection negotiation failed: client connection to 127.27.46.193:45497: connect: Connection refused (error 111)
I20260925 04:37:30.131176 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14771
I20260925 04:37:30.131740 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:30.131387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:45367 (local address 127.0.0.1:50468)
0925 04:37:30.131571 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:30.131626 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.194:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 04:37:30.137377 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14411
2026-09-25T04:37:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:30 dist-test-slave-r8r3 krb5kdc[14946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:30 dist-test-slave-r8r3 krb5kdc[14946](info): set up 2 sockets
Sep 25 04:37:30 dist-test-slave-r8r3 krb5kdc[14946](info): commencing operation
krb5kdc: starting...
W20260925 04:37:31.182161 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:31.305274 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44171
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:44171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:31.416219 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:31.416560 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:31.416637 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:31.420763 14959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:31.420871 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:31.420912 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:31.420948 14959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:31.420989 14959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:31.426353 14959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:44171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:31.427502 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:31.428576 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:31.437323 14964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:31.437331 14967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:31.437331 14965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:31.437419 14959 server_base.cc:1034] running on GCE node
I20260925 04:37:31.438282 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:31.439462 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:31.440670 14959 hybrid_clock.cc:648] HybridClock initialized: now 1790311051440654 us; error 38 us; skew 500 ppm
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:31.443552 14959 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:31.444579 14959 webserver.cc:460] Webserver started at http://127.27.46.254:37159/ using document root <none> and password file <none>
I20260925 04:37:31.445178 14959 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:31.445272 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:31.445568 14959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:31.447706 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "eb6ab1a17b8c41b6b65da65d75d64553"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "41cabe3ff11ef27b5a36034f4ba59e57"
server_key_iv: "da2e695e2064e9b81a82997b7418e6b4"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.448283 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "eb6ab1a17b8c41b6b65da65d75d64553"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "41cabe3ff11ef27b5a36034f4ba59e57"
server_key_iv: "da2e695e2064e9b81a82997b7418e6b4"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.452245 14959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 04:37:31.454873 14974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:31.456276 14959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:37:31.456422 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "eb6ab1a17b8c41b6b65da65d75d64553"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "41cabe3ff11ef27b5a36034f4ba59e57"
server_key_iv: "da2e695e2064e9b81a82997b7418e6b4"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.456547 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:31.470037 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:31.473060 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:31.473286 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:31.480759 14959 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:44171
I20260925 04:37:31.480791 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:44171 every 8 connection(s)
I20260925 04:37:31.481837 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:31.484668 15028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:31.490509 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: Bootstrap starting.
I20260925 04:37:31.491935 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 14959
I20260925 04:37:31.492079 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:31.492374 27835 external_mini_cluster.cc:1428] Setting key 6be09415db34d851701c2965618fb47d
I20260925 04:37:31.493606 15028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:31.494485 15028 log.cc:824] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:31.496527 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: No bootstrap required, opened a new log
I20260925 04:37:31.501066 15028 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } }
I20260925 04:37:31.501307 15028 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:31.501367 15028 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6ab1a17b8c41b6b65da65d75d64553, State: Initialized, Role: FOLLOWER
I20260925 04:37:31.501910 15028 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } }
I20260925 04:37:31.502099 15028 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:31.502172 15028 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:31.502283 15028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:31.503496 15028 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } }
I20260925 04:37:31.503908 15028 leader_election.cc:302] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6ab1a17b8c41b6b65da65d75d64553; no voters:
I20260925 04:37:31.504370 15028 leader_election.cc:288] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:31.504483 15033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:31.504757 15033 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [term 1 LEADER]: Becoming Leader. State: Replica: eb6ab1a17b8c41b6b65da65d75d64553, State: Running, Role: LEADER
I20260925 04:37:31.505096 15033 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } }
I20260925 04:37:31.505838 15028 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:31.506932 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6ab1a17b8c41b6b65da65d75d64553. Latest consensus state: current_term: 1 leader_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } } }
I20260925 04:37:31.507071 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:31.507409 15034 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6ab1a17b8c41b6b65da65d75d64553" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 44171 } } }
I20260925 04:37:31.507704 15034 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:31.508039 15042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:31.509204 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:31.493673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.27.46.254:44171)
0925 04:37:31.494173 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:31.494197 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:31.494236 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:37:31.495062 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:31.495089 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:31.495505 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:31.495860 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:31.496717 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.498246 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.499116 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.499940 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.500099 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:31.502956 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:31.502962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:31.502980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:31.503012 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:31.505046 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:31.505575 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:31.505582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:31.505588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:31.505735 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:31.506078 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:31.506082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:31.506083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:31.506630 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:31.506874 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:31.507726 (+ 852us) server_negotiation.cc:299] Negotiation successful
0925 04:37:31.508095 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":165,"threads_started":1}
I20260925 04:37:31.511389 15042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:31.517158 15042 catalog_manager.cc:1347] Generated new cluster ID: 19b9b643d6004a278d70da329be925a3
I20260925 04:37:31.517244 15042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:31.528040 15042 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:31.529304 15042 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:31.539444 15042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: Generated new TSK 0
I20260925 04:37:31.540216 15042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
Entry for principal kudu/127.27.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab.
I20260925 04:37:31.573215 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:31.688189 15054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:31.688525 15054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:31.688606 15054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:31.692716 15054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:31.692833 15054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:31.692931 15054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:31.698441 15054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.193.keytab
--principal=kudu/127.27.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:31.699656 15054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:31.700740 15054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:31.711149 15062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:31.711190 15059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:31.711351 15054 server_base.cc:1034] running on GCE node
W20260925 04:37:31.711190 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:31.711941 15054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:31.712590 15054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:31.713778 15054 hybrid_clock.cc:648] HybridClock initialized: now 1790311051713759 us; error 30 us; skew 500 ppm
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:31.716578 15054 init.cc:377] Logged in from keytab as kudu/127.27.46.193@KRBTEST.COM (short username kudu)
I20260925 04:37:31.717584 15054 webserver.cc:460] Webserver started at http://127.27.46.193:44649/ using document root <none> and password file <none>
I20260925 04:37:31.718182 15054 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:31.718245 15054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:31.718510 15054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:31.720722 15054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "6f50963f26a140b59014a73722c7c595"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "090a5eed05b390424ea566338fbf769a"
server_key_iv: "3a20c67e1f4b6d139a10f8817f893e9f"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.721293 15054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "6f50963f26a140b59014a73722c7c595"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "090a5eed05b390424ea566338fbf769a"
server_key_iv: "3a20c67e1f4b6d139a10f8817f893e9f"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.725131 15054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:37:31.727770 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:31.729019 15054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:31.729162 15054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "6f50963f26a140b59014a73722c7c595"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "090a5eed05b390424ea566338fbf769a"
server_key_iv: "3a20c67e1f4b6d139a10f8817f893e9f"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.729288 15054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:31.762645 15054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:31.768085 15054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:31.768381 15054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:31.769110 15054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:31.770253 15054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:31.770337 15054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:31.770413 15054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:31.770465 15054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:31.779932 15054 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:44717
I20260925 04:37:31.779953 15185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:44717 every 8 connection(s)
I20260925 04:37:31.780912 15054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:31.781294 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15054
I20260925 04:37:31.781448 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:31.781805 27835 external_mini_cluster.cc:1428] Setting key 232074c72f99ba68648f4c19a5955cb0
WARNING: no policy specified for kudu/127.27.46.194@KRBTEST.COM; defaulting to no policy
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.193@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.194@KRBTEST.COM" created.
I20260925 04:37:31.796278 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:31.782859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:40097 (local address 127.27.46.254:44171)
0925 04:37:31.783015 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:31.783020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:31.784028 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 04:37:31.785409 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:31.785415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:31.785487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:31.785614 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:31.786822 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.787757 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.788919 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.789390 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.789557 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:31.792309 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:31.792312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:31.792314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:31.792341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:31.793831 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:31.794379 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:31.794383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:31.794385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:31.794487 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:31.794855 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:31.794857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:31.794859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:31.795056 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:31.795147 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:31.795870 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 04:37:31.796036 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:37:31.797166 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:31.783346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44171 (local address 127.27.46.193:40097)
0925 04:37:31.783856 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:31.783912 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:31.785191 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:31.785790 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:31.785797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:31.786162 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:31.786651 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:31.786665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.787897 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.787900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:31.788784 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:31.788793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:31.789615 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:31.789619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:31.789782 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:31.790393 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:31.790409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:31.792101 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:31.793947 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:31.793952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:31.793965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:31.794264 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:31.794575 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:31.794577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:31.794579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:31.794748 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:31.795169 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:31.795178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:31.795558 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 04:37:31.795798 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 04:37:31.796208 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":133,"threads_started":1}
I20260925 04:37:31.798626 15186 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44171
I20260925 04:37:31.798899 15186 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:31.799575 15186 heartbeater.cc:511] Master 127.27.46.254:44171 requested a full tablet report, sending...
I20260925 04:37:31.801292 14991 ts_manager.cc:194] Registered new tserver with Master: 6f50963f26a140b59014a73722c7c595 (127.27.46.193:44717)
I20260925 04:37:31.802508 14991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.193@KRBTEST.COM'} at 127.27.46.193:40097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
Entry for principal kudu/127.27.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab.
I20260925 04:37:31.812090 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:31.927354 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:31.927640 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:31.927716 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:31.931703 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:31.931789 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:31.931897 15191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:31.937199 15191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.194.keytab
--principal=kudu/127.27.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:31.938278 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:31.939306 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:31.948616 15199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:31.948616 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:31.948664 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:31.948841 15191 server_base.cc:1034] running on GCE node
I20260925 04:37:31.949297 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:31.949954 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:31.951136 15191 hybrid_clock.cc:648] HybridClock initialized: now 1790311051951125 us; error 32 us; skew 500 ppm
Sep 25 04:37:31 dist-test-slave-r8r3 krb5kdc[14946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:31.953806 15191 init.cc:377] Logged in from keytab as kudu/127.27.46.194@KRBTEST.COM (short username kudu)
I20260925 04:37:31.954730 15191 webserver.cc:460] Webserver started at http://127.27.46.194:41287/ using document root <none> and password file <none>
I20260925 04:37:31.955271 15191 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:31.955369 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:31.955602 15191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:31.957762 15191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "ec49b468fda449ebb89212249b5ceaf4"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "fc62de62958c9202c3179427074c1d26"
server_key_iv: "8aa081946009dfa5ea01e0ff682b18ec"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.958331 15191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "ec49b468fda449ebb89212249b5ceaf4"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "fc62de62958c9202c3179427074c1d26"
server_key_iv: "8aa081946009dfa5ea01e0ff682b18ec"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.962076 15191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:37:31.964421 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:31.965700 15191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:31.965847 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "ec49b468fda449ebb89212249b5ceaf4"
format_stamp: "Formatted at 2026-09-25 04:37:31 on dist-test-slave-r8r3"
server_key: "fc62de62958c9202c3179427074c1d26"
server_key_iv: "8aa081946009dfa5ea01e0ff682b18ec"
server_key_version: "encryptionkey@0"
I20260925 04:37:31.965958 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:31.995157 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:31.998624 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:31.998900 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:31.999517 15191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:32.000358 15191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:32.000432 15191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.000510 15191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:32.000556 15191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.009999 15191 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:36207
I20260925 04:37:32.010015 15321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:36207 every 8 connection(s)
I20260925 04:37:32.010924 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:32.019315 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15191
I20260925 04:37:32.019411 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:32.019686 27835 external_mini_cluster.cc:1428] Setting key d648f448bfa6b828e93dbe0d2d66370c
Sep 25 04:37:32 dist-test-slave-r8r3 krb5kdc[14946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311051, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.194@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:32.025604 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.012715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:36595 (local address 127.27.46.254:44171)
0925 04:37:32.012853 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.012857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.013761 (+ 904us) server_negotiation.cc:407] Connection header received
0925 04:37:32.015037 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.015039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.015085 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.015157 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:32.016219 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.016710 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.017857 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.018063 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.018139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.020595 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:32.020599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:32.020601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:32.020625 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:32.023072 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.023735 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.023781 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.023784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.023920 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.024387 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.024394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.024399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.024666 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:32.024769 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.025217 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.025347 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:37:32.026363 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.013116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44171 (local address 127.27.46.194:36595)
0925 04:37:32.013617 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.013661 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.014852 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.015314 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.015320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.015660 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:32.016100 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.016113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.016811 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.016813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.017756 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.017765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.018188 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.018191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.018335 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.018847 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:32.018863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:32.020428 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:32.023186 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:32.023192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:32.023206 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:32.023591 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:32.024074 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:32.024080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:32.024083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:32.024280 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:32.024775 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:32.024780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:32.024992 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.025137 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.025370 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260925 04:37:32.027635 15322 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44171
I20260925 04:37:32.027884 15322 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:32.028364 15322 heartbeater.cc:511] Master 127.27.46.254:44171 requested a full tablet report, sending...
I20260925 04:37:32.029458 14991 ts_manager.cc:194] Registered new tserver with Master: ec49b468fda449ebb89212249b5ceaf4 (127.27.46.194:36207)
WARNING: no policy specified for kudu/127.27.46.195@KRBTEST.COM; defaulting to no policy
I20260925 04:37:32.030323 14991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.194@KRBTEST.COM'} at 127.27.46.194:36595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
Entry for principal kudu/127.27.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab.
I20260925 04:37:32.048162 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:32.156705 15327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:32.156953 15327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:32.156996 15327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:32.161662 15327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:32.161726 15327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:32.161834 15327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:32.167397 15327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.195.keytab
--principal=kudu/127.27.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:44171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:32.168509 15327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:32.169646 15327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:32.179229 15333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:32.179292 15332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:32.179275 15327 server_base.cc:1034] running on GCE node
W20260925 04:37:32.179205 15335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:32.179888 15327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:32.180480 15327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:32.181643 15327 hybrid_clock.cc:648] HybridClock initialized: now 1790311052181633 us; error 50 us; skew 500 ppm
Sep 25 04:37:32 dist-test-slave-r8r3 krb5kdc[14946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:32.184245 15327 init.cc:377] Logged in from keytab as kudu/127.27.46.195@KRBTEST.COM (short username kudu)
I20260925 04:37:32.185168 15327 webserver.cc:460] Webserver started at http://127.27.46.195:45049/ using document root <none> and password file <none>
I20260925 04:37:32.185812 15327 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:32.185909 15327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:32.186139 15327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:32.188184 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "00207c0efa3442c6ae361d7eb490d308"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "f139463301281ca4c1403a5a613aed8d"
server_key_iv: "bdb534d02db366f165c96d866a69f53f"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.188782 15327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "00207c0efa3442c6ae361d7eb490d308"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "f139463301281ca4c1403a5a613aed8d"
server_key_iv: "bdb534d02db366f165c96d866a69f53f"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.192595 15327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 04:37:32.195214 15342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.196414 15327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:37:32.196550 15327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "00207c0efa3442c6ae361d7eb490d308"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "f139463301281ca4c1403a5a613aed8d"
server_key_iv: "bdb534d02db366f165c96d866a69f53f"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.196666 15327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:32.228206 15327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:32.231693 15327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:32.231916 15327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:32.232547 15327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:32.233454 15327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:32.233544 15327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.233635 15327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:32.233685 15327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.242507 15327 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:34169
I20260925 04:37:32.242524 15456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:34169 every 8 connection(s)
I20260925 04:37:32.243434 15327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:32.244863 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15327
I20260925 04:37:32.245009 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:32.245352 27835 external_mini_cluster.cc:1428] Setting key db136c192b02368eeb6a10704b10c7a7
Sep 25 04:37:32 dist-test-slave-r8r3 krb5kdc[14946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.195@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:32.256537 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.245343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:54641 (local address 127.27.46.254:44171)
0925 04:37:32.245517 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.245524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.246492 (+ 968us) server_negotiation.cc:407] Connection header received
0925 04:37:32.248086 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.248089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.248136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.248203 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:32.249314 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.249912 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.251005 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.251208 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.251292 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.253633 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:32.253636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:32.253638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:32.253661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:32.254860 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.255310 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.255313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.255314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.255383 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.255659 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.255661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.255662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.255831 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:32.255904 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.256318 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.256406 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:37:32.257336 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.245825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:44171 (local address 127.27.46.195:54641)
0925 04:37:32.246331 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.246391 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.247900 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.248334 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.248340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.248679 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:32.249202 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.249214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.250014 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.250016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.250903 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.250912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.251330 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.251332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.251478 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.251965 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:37:32.251980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:32.253427 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:32.254959 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:32.254963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:32.254975 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:32.255220 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:32.255457 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:37:32.255459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:37:32.255460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:37:32.255585 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:37:32.255914 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:32.255918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:32.256126 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.256283 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.256523 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260925 04:37:32.258682 15457 heartbeater.cc:347] Connected to a master server at 127.27.46.254:44171
I20260925 04:37:32.258950 15457 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:32.259495 15457 heartbeater.cc:511] Master 127.27.46.254:44171 requested a full tablet report, sending...
I20260925 04:37:32.260488 14991 ts_manager.cc:194] Registered new tserver with Master: 00207c0efa3442c6ae361d7eb490d308 (127.27.46.195:34169)
I20260925 04:37:32.261005 14991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.46.195@KRBTEST.COM'} at 127.27.46.195:54641
I20260925 04:37:32.271180 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:32.280527 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.273179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.27.46.254:44171)
0925 04:37:32.273320 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.273324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.273531 (+ 207us) server_negotiation.cc:407] Connection header received
0925 04:37:32.273653 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.273655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.273697 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.273772 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:32.274473 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.274983 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.275768 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.275969 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.276037 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.277304 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:32.277306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:32.277308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:32.277331 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:32.278826 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.279365 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.279367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.279368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.279437 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:32.279748 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:32.279750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:32.279751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:32.279968 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:32.280072 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.280313 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.280405 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:37:32.283636 14991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:32.286600 14991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:32.301277 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.295421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:34169 (local address 127.0.0.1:48976)
0925 04:37:32.295884 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.295931 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.296058 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.297241 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.297244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.297280 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:32.297389 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.297395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298560 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.298566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.299762 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.299772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.300731 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.300735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.300857 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.300898 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.301007 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.301119 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":74,"threads_started":1}
I20260925 04:37:32.301537 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.295496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44717 (local address 127.0.0.1:58282)
0925 04:37:32.296280 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.296293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.296373 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.297101 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.297107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.297129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:32.297281 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.297295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298932 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.298938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.299932 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.299943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.301009 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.301013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.301102 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.301122 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.301177 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.301228 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":14,"thread_start_us":122,"threads_started":1}
I20260925 04:37:32.301837 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.296225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.27.46.193:44717)
0925 04:37:32.296585 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.296600 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.296622 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:37:32.296701 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.296707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.296931 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.297087 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:32.297422 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298823 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.300062 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.300878 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.301067 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.301229 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.301283 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.301698 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260925 04:37:32.302013 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.295537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.27.46.195:34169)
0925 04:37:32.296656 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.296667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.296685 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:37:32.296744 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.296749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.296931 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.297087 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:32.297539 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298424 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.299889 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.300608 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.300731 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.300877 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.301778 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.301856 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":110,"threads_started":1}
I20260925 04:37:32.302734 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.296144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:36207 (local address 127.0.0.1:60156)
0925 04:37:32.296661 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.296675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.296761 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.297374 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.297376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.297389 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:32.297469 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.297476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298861 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.298867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.301055 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.301065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.302347 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.302351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.302444 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.302464 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.302539 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.302598 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":55,"thread_start_us":84,"threads_started":1}
I20260925 04:37:32.302955 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.296517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.27.46.194:36207)
0925 04:37:32.296949 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.296958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.296974 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:37:32.297062 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.297068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.297239 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.297390 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:32.297587 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.298699 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.301183 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.302050 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.302163 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.302286 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.302620 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.302696 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":95,"threads_started":1}
I20260925 04:37:32.304831 15118 tablet_service.cc:1467] Processing CreateTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 (DEFAULT_TABLE table=test-table [id=5ddc32d64771415f846636c6897ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:32.305384 15390 tablet_service.cc:1467] Processing CreateTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 (DEFAULT_TABLE table=test-table [id=5ddc32d64771415f846636c6897ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:32.305384 15254 tablet_service.cc:1467] Processing CreateTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 (DEFAULT_TABLE table=test-table [id=5ddc32d64771415f846636c6897ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:32.305953 15118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01073c88ea7f4a679ebe84dcfecfd4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:32.306362 15390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01073c88ea7f4a679ebe84dcfecfd4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:32.306362 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01073c88ea7f4a679ebe84dcfecfd4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:32.313056 15474 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: Bootstrap starting.
I20260925 04:37:32.313767 15475 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Bootstrap starting.
I20260925 04:37:32.315138 15474 tablet_bootstrap.cc:654] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:32.315428 15475 tablet_bootstrap.cc:654] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:32.315900 15474 log.cc:824] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:32.316614 15475 log.cc:824] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:32.316721 15476 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Bootstrap starting.
I20260925 04:37:32.318485 15476 tablet_bootstrap.cc:654] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:32.318497 15475 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: No bootstrap required, opened a new log
I20260925 04:37:32.318717 15475 ts_tablet_manager.cc:1397] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:37:32.319532 15476 log.cc:824] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:32.320986 15474 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: No bootstrap required, opened a new log
I20260925 04:37:32.321209 15474 ts_tablet_manager.cc:1397] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260925 04:37:32.321436 15476 tablet_bootstrap.cc:492] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: No bootstrap required, opened a new log
I20260925 04:37:32.321642 15476 ts_tablet_manager.cc:1397] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 04:37:32.322274 15475 raft_consensus.cc:348] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.322553 15475 raft_consensus.cc:374] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:32.322621 15475 raft_consensus.cc:729] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec49b468fda449ebb89212249b5ceaf4, State: Initialized, Role: FOLLOWER
I20260925 04:37:32.323086 15475 consensus_queue.cc:260] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.324154 15475 ts_tablet_manager.cc:1428] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 04:37:32.324544 15322 heartbeater.cc:503] Master 127.27.46.254:44171 was elected leader, sending a full tablet report...
I20260925 04:37:32.325438 15476 raft_consensus.cc:348] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.325529 15474 raft_consensus.cc:348] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.325791 15476 raft_consensus.cc:374] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:32.325814 15474 raft_consensus.cc:374] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:32.325855 15476 raft_consensus.cc:729] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00207c0efa3442c6ae361d7eb490d308, State: Initialized, Role: FOLLOWER
I20260925 04:37:32.325870 15474 raft_consensus.cc:729] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f50963f26a140b59014a73722c7c595, State: Initialized, Role: FOLLOWER
I20260925 04:37:32.326282 15476 consensus_queue.cc:260] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.326321 15474 consensus_queue.cc:260] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.326941 15457 heartbeater.cc:503] Master 127.27.46.254:44171 was elected leader, sending a full tablet report...
I20260925 04:37:32.327291 15476 ts_tablet_manager.cc:1428] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 04:37:32.327298 15474 ts_tablet_manager.cc:1428] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 04:37:32.327642 15186 heartbeater.cc:503] Master 127.27.46.254:44171 was elected leader, sending a full tablet report...
I20260925 04:37:32.368875 15482 raft_consensus.cc:482] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:32.369154 15482 raft_consensus.cc:504] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.370514 15482 leader_election.cc:288] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00207c0efa3442c6ae361d7eb490d308 (127.27.46.195:34169), ec49b468fda449ebb89212249b5ceaf4 (127.27.46.194:36207)
I20260925 04:37:32.375941 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.370870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:37551 (local address 127.27.46.195:34169)
0925 04:37:32.371045 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.371050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.371178 (+ 128us) server_negotiation.cc:407] Connection header received
0925 04:37:32.371432 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.371435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.371515 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.371571 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:32.372052 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.372915 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.374712 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.375333 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.375461 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.375515 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.375723 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.375805 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:37:32.375948 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.370775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:34169 (local address 127.27.46.193:37551)
0925 04:37:32.371087 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.371105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.371280 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.371748 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.371752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.371768 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:32.371900 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.371930 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.373046 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.373051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.374573 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.374584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.375472 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.375476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.375578 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.375605 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.375683 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.375755 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":142,"thread_start_us":137,"threads_started":1}
I20260925 04:37:32.376075 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.371328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:38395 (local address 127.27.46.194:36207)
0925 04:37:32.371470 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.371475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.371691 (+ 216us) server_negotiation.cc:407] Connection header received
0925 04:37:32.371822 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.371824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.371868 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.371927 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:32.372213 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.372971 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.374712 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.375343 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.375461 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.375515 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.375869 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.375949 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:37:32.376286 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.371211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:36207 (local address 127.27.46.193:38395)
0925 04:37:32.371597 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.371611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.371715 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.371963 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.371969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.371991 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:32.372109 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.372114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.373074 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.373079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.374573 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.374585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.375606 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.375609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.375731 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.375768 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.375846 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.375912 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":58,"thread_start_us":89,"threads_started":1}
I20260925 04:37:32.376856 15410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01073c88ea7f4a679ebe84dcfecfd4f9" candidate_uuid: "6f50963f26a140b59014a73722c7c595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00207c0efa3442c6ae361d7eb490d308" is_pre_election: true
I20260925 04:37:32.376856 15274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01073c88ea7f4a679ebe84dcfecfd4f9" candidate_uuid: "6f50963f26a140b59014a73722c7c595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec49b468fda449ebb89212249b5ceaf4" is_pre_election: true
I20260925 04:37:32.377189 15410 raft_consensus.cc:2454] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f50963f26a140b59014a73722c7c595 in term 0.
I20260925 04:37:32.377189 15274 raft_consensus.cc:2454] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f50963f26a140b59014a73722c7c595 in term 0.
I20260925 04:37:32.377707 15070 leader_election.cc:302] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f50963f26a140b59014a73722c7c595, ec49b468fda449ebb89212249b5ceaf4; no voters:
I20260925 04:37:32.378011 15482 raft_consensus.cc:2781] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:32.378118 15482 raft_consensus.cc:482] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:32.378166 15482 raft_consensus.cc:3037] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:32.379271 15482 raft_consensus.cc:504] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.379633 15482 leader_election.cc:288] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [CANDIDATE]: Term 1 election: Requested vote from peers 00207c0efa3442c6ae361d7eb490d308 (127.27.46.195:34169), ec49b468fda449ebb89212249b5ceaf4 (127.27.46.194:36207)
I20260925 04:37:32.380079 15410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01073c88ea7f4a679ebe84dcfecfd4f9" candidate_uuid: "6f50963f26a140b59014a73722c7c595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00207c0efa3442c6ae361d7eb490d308"
I20260925 04:37:32.380097 15274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01073c88ea7f4a679ebe84dcfecfd4f9" candidate_uuid: "6f50963f26a140b59014a73722c7c595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec49b468fda449ebb89212249b5ceaf4"
I20260925 04:37:32.380220 15410 raft_consensus.cc:3037] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:32.380220 15274 raft_consensus.cc:3037] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:32.381587 15410 raft_consensus.cc:2454] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f50963f26a140b59014a73722c7c595 in term 1.
I20260925 04:37:32.381587 15274 raft_consensus.cc:2454] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f50963f26a140b59014a73722c7c595 in term 1.
I20260925 04:37:32.381897 15070 leader_election.cc:302] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f50963f26a140b59014a73722c7c595, ec49b468fda449ebb89212249b5ceaf4; no voters:
I20260925 04:37:32.382069 15482 raft_consensus.cc:2781] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:32.382328 15482 raft_consensus.cc:686] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [term 1 LEADER]: Becoming Leader. State: Replica: 6f50963f26a140b59014a73722c7c595, State: Running, Role: LEADER
I20260925 04:37:32.382700 15482 consensus_queue.cc:237] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } }
I20260925 04:37:32.386153 14990 catalog_manager.cc:5495] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f50963f26a140b59014a73722c7c595 (127.27.46.193). New cstate: current_term: 1 leader_uuid: "6f50963f26a140b59014a73722c7c595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f50963f26a140b59014a73722c7c595" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 44717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 } health_report { overall_health: UNKNOWN } } }
I20260925 04:37:32.398699 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.395030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.27.46.193:44717)
0925 04:37:32.395168 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.395173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.395198 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:37:32.395371 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.395374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.395428 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.395538 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:32.395793 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.396309 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.397205 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.397424 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.397578 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.397876 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:32.398296 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:32.398430 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.398490 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.398550 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:37:32.404640 15274 raft_consensus.cc:1261] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 1 FOLLOWER]: Refusing update from remote peer 6f50963f26a140b59014a73722c7c595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:32.404641 15410 raft_consensus.cc:1261] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 1 FOLLOWER]: Refusing update from remote peer 6f50963f26a140b59014a73722c7c595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:32.405319 15482 consensus_queue.cc:1035] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec49b468fda449ebb89212249b5ceaf4" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 36207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:32.405498 15485 consensus_queue.cc:1035] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 6f50963f26a140b59014a73722c7c595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00207c0efa3442c6ae361d7eb490d308" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:32.412873 15488 mvcc.cc:205] Tried to move back new op lower bound from 7333114070642282496 to 7333114070562717696. Current Snapshot: MvccSnapshot[applied={T|T < 7333114070642282496}]
I20260925 04:37:32.413756 15490 mvcc.cc:205] Tried to move back new op lower bound from 7333114070642282496 to 7333114070562717696. Current Snapshot: MvccSnapshot[applied={T|T < 7333114070642282496}]
I20260925 04:37:32.421950 14991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:32.422155 14991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:32.424789 14991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb6ab1a17b8c41b6b65da65d75d64553: Sending DeleteTablet for 3 replicas of tablet 01073c88ea7f4a679ebe84dcfecfd4f9
I20260925 04:37:32.425640 15118 tablet_service.cc:1514] Processing DeleteTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:58282
I20260925 04:37:32.425870 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:60156
I20260925 04:37:32.426215 15390 tablet_service.cc:1514] Processing DeleteTablet for tablet 01073c88ea7f4a679ebe84dcfecfd4f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:32 UTC) from {username='kudu', principal='kudu/127.27.46.254@KRBTEST.COM'} at 127.0.0.1:48976
I20260925 04:37:32.426384 15499 tablet_replica.cc:330] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: stopping tablet replica
I20260925 04:37:32.426671 15500 tablet_replica.cc:330] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: stopping tablet replica
I20260925 04:37:32.426728 15499 raft_consensus.cc:2229] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:32.426855 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15054
I20260925 04:37:32.427927 15499 raft_consensus.cc:2258] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:32.427983 15500 raft_consensus.cc:2229] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:32.428634 15500 raft_consensus.cc:2258] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:32.430104 15500 ts_tablet_manager.cc:1905] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:32.430141 15499 ts_tablet_manager.cc:1905] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:32.432381 15500 ts_tablet_manager.cc:1918] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:32.432509 15500 log.cc:1196] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/01073c88ea7f4a679ebe84dcfecfd4f9
I20260925 04:37:32.432798 15500 ts_tablet_manager.cc:1939] T 01073c88ea7f4a679ebe84dcfecfd4f9 P 00207c0efa3442c6ae361d7eb490d308: Deleting consensus metadata
I20260925 04:37:32.433516 15499 ts_tablet_manager.cc:1918] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:32.433620 15499 log.cc:1196] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/01073c88ea7f4a679ebe84dcfecfd4f9
I20260925 04:37:32.433961 15499 ts_tablet_manager.cc:1939] T 01073c88ea7f4a679ebe84dcfecfd4f9 P ec49b468fda449ebb89212249b5ceaf4: Deleting consensus metadata
I20260925 04:37:32.434077 14975 catalog_manager.cc:4841] TS 00207c0efa3442c6ae361d7eb490d308 (127.27.46.195:34169): tablet 01073c88ea7f4a679ebe84dcfecfd4f9 (table test-table [id=5ddc32d64771415f846636c6897ea869]) successfully deleted
I20260925 04:37:32.435134 14975 catalog_manager.cc:4841] TS ec49b468fda449ebb89212249b5ceaf4 (127.27.46.194:36207): tablet 01073c88ea7f4a679ebe84dcfecfd4f9 (table test-table [id=5ddc32d64771415f846636c6897ea869]) successfully deleted
W20260925 04:37:32.435938 14976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:44717 (error 108)
I20260925 04:37:32.436087 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15191
I20260925 04:37:32.438488 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.438218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:44717 (local address 127.0.0.1:58298)
0925 04:37:32.438345 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.438414 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:37:32.438635 14976 catalog_manager.cc:4584] TS 6f50963f26a140b59014a73722c7c595 (127.27.46.193:44717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01073c88ea7f4a679ebe84dcfecfd4f9: Network error: Client connection negotiation failed: client connection to 127.27.46.193:44717: connect: Connection refused (error 111)
I20260925 04:37:32.442615 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15327
I20260925 04:37:32.448769 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 14959
2026-09-25T04:37:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2312 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T04:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:32Z Disabled control of system clock
I20260925 04:37:32.480494 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43711
--webserver_interface=127.27.46.254
--webserver_port=0
--builtin_ntp_servers=127.27.46.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:32.593590 15505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:32.593876 15505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:32.593912 15505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:32.597954 15505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:32.598009 15505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:32.598024 15505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:32.598042 15505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:32.598057 15505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:32.602757 15505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:43711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:32.603852 15505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:32.605217 15505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:32.613842 15513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:32.613901 15510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:32.614023 15505 server_base.cc:1034] running on GCE node
W20260925 04:37:32.614216 15511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:32.614846 15505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:32.616009 15505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:32.617195 15505 hybrid_clock.cc:648] HybridClock initialized: now 1790311052617182 us; error 37 us; skew 500 ppm
I20260925 04:37:32.619189 15505 webserver.cc:460] Webserver started at http://127.27.46.254:37943/ using document root <none> and password file <none>
I20260925 04:37:32.619799 15505 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:32.619899 15505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:32.620137 15505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:32.622292 15505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "17867fd422494d49b6951f83d0b0e2d6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "e07d1e6c40d48a41c46f451b27d87e0a"
server_key_iv: "a2da86ae8561878f3185b39a292f5071"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.622861 15505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "17867fd422494d49b6951f83d0b0e2d6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "e07d1e6c40d48a41c46f451b27d87e0a"
server_key_iv: "a2da86ae8561878f3185b39a292f5071"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.626605 15505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:37:32.629016 15519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.630297 15505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:32.630438 15505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "17867fd422494d49b6951f83d0b0e2d6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "e07d1e6c40d48a41c46f451b27d87e0a"
server_key_iv: "a2da86ae8561878f3185b39a292f5071"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.630553 15505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:32.644795 15505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:32.645627 15505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:32.645828 15505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:32.652917 15505 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:43711
I20260925 04:37:32.652942 15573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:43711 every 8 connection(s)
I20260925 04:37:32.653998 15505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:32.656427 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15505
I20260925 04:37:32.656585 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:32.656903 27835 external_mini_cluster.cc:1428] Setting key ca5734466afea06bee456f310df25420
I20260925 04:37:32.657394 15574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:32.664623 15574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: Bootstrap starting.
I20260925 04:37:32.666375 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.658350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.27.46.254:43711)
0925 04:37:32.659002 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.659018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.659054 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:37:32.659707 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.659722 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.660105 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.660382 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:32.660757 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.661799 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.662965 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.663439 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.663556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.664496 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:32.664501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:32.664514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:32.664999 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:32.665057 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:32.665078 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:32.665153 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.665324 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.665663 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":154,"threads_started":1}
I20260925 04:37:32.667116 15574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:32.667886 15574 log.cc:824] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:32.669970 15574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: No bootstrap required, opened a new log
I20260925 04:37:32.673406 15574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } }
I20260925 04:37:32.673708 15574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:32.673794 15574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17867fd422494d49b6951f83d0b0e2d6, State: Initialized, Role: FOLLOWER
I20260925 04:37:32.674294 15574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } }
I20260925 04:37:32.674459 15574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:32.674535 15574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:32.674667 15574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:32.675625 15574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } }
I20260925 04:37:32.675932 15574 leader_election.cc:302] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17867fd422494d49b6951f83d0b0e2d6; no voters:
I20260925 04:37:32.676208 15574 leader_election.cc:288] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:32.676373 15580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:32.676648 15580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [term 1 LEADER]: Becoming Leader. State: Replica: 17867fd422494d49b6951f83d0b0e2d6, State: Running, Role: LEADER
I20260925 04:37:32.676971 15580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } }
I20260925 04:37:32.677523 15574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:32.678651 15581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17867fd422494d49b6951f83d0b0e2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } } }
I20260925 04:37:32.678781 15581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:32.679101 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17867fd422494d49b6951f83d0b0e2d6. Latest consensus state: current_term: 1 leader_uuid: "17867fd422494d49b6951f83d0b0e2d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17867fd422494d49b6951f83d0b0e2d6" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43711 } } }
I20260925 04:37:32.679221 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:32.679517 15589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:32.682983 15589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:32.688295 15589 catalog_manager.cc:1347] Generated new cluster ID: 34ca4f94794d4181a9240f61b50ffa88
I20260925 04:37:32.688374 15589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:32.702907 15589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:32.704218 15589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:32.724054 15589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: Generated new TSK 0
I20260925 04:37:32.724838 15589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:32.730132 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.193:0
--local_ip_for_outbound_sockets=127.27.46.193
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--builtin_ntp_servers=127.27.46.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:37:32.850632 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:32.850944 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:32.851014 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:32.855058 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:32.855142 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:32.855233 15599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.193
I20260925 04:37:32.860280 15599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.46.193
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.193
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:32.861609 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:32.863106 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:37:32.873289 15599 server_base.cc:1034] running on GCE node
W20260925 04:37:32.873245 15607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:32.873247 15605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:32.873272 15604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:32.874069 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:32.874801 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:32.875968 15599 hybrid_clock.cc:648] HybridClock initialized: now 1790311052875953 us; error 27 us; skew 500 ppm
I20260925 04:37:32.877981 15599 webserver.cc:460] Webserver started at http://127.27.46.193:43481/ using document root <none> and password file <none>
I20260925 04:37:32.878583 15599 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:32.878690 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:32.878930 15599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:32.881103 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data/instance:
uuid: "73b16fa223ca46f08a86065ee0b95fa6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "04ecac05d0901c4cd12e4300af8453f8"
server_key_iv: "2b97776c7f8eddf7ba2cc1ef41dc4252"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.881742 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance:
uuid: "73b16fa223ca46f08a86065ee0b95fa6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "04ecac05d0901c4cd12e4300af8453f8"
server_key_iv: "2b97776c7f8eddf7ba2cc1ef41dc4252"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.885775 15599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:37:32.888372 15613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.889760 15599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:32.889909 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal
uuid: "73b16fa223ca46f08a86065ee0b95fa6"
format_stamp: "Formatted at 2026-09-25 04:37:32 on dist-test-slave-r8r3"
server_key: "04ecac05d0901c4cd12e4300af8453f8"
server_key_iv: "2b97776c7f8eddf7ba2cc1ef41dc4252"
server_key_version: "encryptionkey@0"
I20260925 04:37:32.890033 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:32.907500 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:32.908250 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:32.908468 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:32.909101 15599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:32.910111 15599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:32.910182 15599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.910254 15599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:32.910296 15599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:32.919430 15599 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.193:38915
I20260925 04:37:32.919456 15729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.193:38915 every 8 connection(s)
I20260925 04:37:32.920397 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/data/info.pb
I20260925 04:37:32.927907 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15599
I20260925 04:37:32.928047 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-0/wal/instance
I20260925 04:37:32.928416 27835 external_mini_cluster.cc:1428] Setting key 2ec6862ffaba3666fb04692a85ae79d2
I20260925 04:37:32.929977 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.922262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.193:50855 (local address 127.27.46.254:43711)
0925 04:37:32.922418 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:32.922423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:32.923540 (+ 1117us) server_negotiation.cc:407] Connection header received
0925 04:37:32.924892 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:32.924894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:32.924956 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:32.925032 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:32.926008 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.926639 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.927810 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.928155 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.928289 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.929432 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:32.929436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:32.929439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:32.929452 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:32.929480 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:32.929488 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:32.929593 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:32.929750 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 04:37:32.929832 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:37:32.930399 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.194:0
--local_ip_for_outbound_sockets=127.27.46.194
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--builtin_ntp_servers=127.27.46.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:32.930910 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:32.922721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:43711 (local address 127.27.46.193:50855)
0925 04:37:32.923387 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:32.923428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:32.924704 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:32.925215 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:32.925222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:32.925307 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:32.925866 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.925881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.926765 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.926767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.927685 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:32.927694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:32.928310 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:32.928313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:32.928478 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:32.929152 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:32.929154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:32.929201 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:32.929263 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:32.929594 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:32.929607 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:32.929613 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:37:32.929750 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 04:37:32.930032 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260925 04:37:32.932171 15730 heartbeater.cc:347] Connected to a master server at 127.27.46.254:43711
I20260925 04:37:32.932437 15730 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:32.933075 15730 heartbeater.cc:511] Master 127.27.46.254:43711 requested a full tablet report, sending...
I20260925 04:37:32.934708 15536 ts_manager.cc:194] Registered new tserver with Master: 73b16fa223ca46f08a86065ee0b95fa6 (127.27.46.193:38915)
I20260925 04:37:32.935948 15536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.193:50855
W20260925 04:37:33.043781 15733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:33.044049 15733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:33.044117 15733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:33.048202 15733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:33.048264 15733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:33.048367 15733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.194
I20260925 04:37:33.053149 15733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.46.194
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.194
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:33.054298 15733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:33.055714 15733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:33.065786 15738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:33.065825 15741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:33.065819 15733 server_base.cc:1034] running on GCE node
W20260925 04:37:33.065963 15739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:33.066475 15733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:33.067138 15733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:33.068333 15733 hybrid_clock.cc:648] HybridClock initialized: now 1790311053068320 us; error 30 us; skew 500 ppm
I20260925 04:37:33.070358 15733 webserver.cc:460] Webserver started at http://127.27.46.194:37087/ using document root <none> and password file <none>
I20260925 04:37:33.070966 15733 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:33.071060 15733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:33.071300 15733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:33.073369 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data/instance:
uuid: "f8915fbf281f4c73b6895132871edeaa"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "6830fadbf94758accd73507d0b2752b8"
server_key_iv: "43606e5e1e8584e322dadb2e7b311f2b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.074013 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance:
uuid: "f8915fbf281f4c73b6895132871edeaa"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "6830fadbf94758accd73507d0b2752b8"
server_key_iv: "43606e5e1e8584e322dadb2e7b311f2b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.077837 15733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:37:33.080437 15747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.081751 15733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:33.081893 15733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal
uuid: "f8915fbf281f4c73b6895132871edeaa"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "6830fadbf94758accd73507d0b2752b8"
server_key_iv: "43606e5e1e8584e322dadb2e7b311f2b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.082008 15733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:33.098780 15733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:33.099608 15733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:33.099828 15733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:33.100443 15733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:33.101459 15733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:33.101552 15733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.101650 15733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:33.101702 15733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.111079 15733 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.194:41125
I20260925 04:37:33.111111 15861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.194:41125 every 8 connection(s)
I20260925 04:37:33.112048 15733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/data/info.pb
I20260925 04:37:33.117800 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15733
I20260925 04:37:33.117951 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal/instance
I20260925 04:37:33.118253 27835 external_mini_cluster.cc:1428] Setting key 421ad0f1d36d7286e7597a57210d7892
I20260925 04:37:33.120501 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.46.195:0
--local_ip_for_outbound_sockets=127.27.46.195
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--builtin_ntp_servers=127.27.46.212:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:37:33.121840 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.113873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:37635 (local address 127.27.46.254:43711)
0925 04:37:33.114016 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.114020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.115073 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 04:37:33.116421 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.116424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.116471 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.116544 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:33.117560 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.118208 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.119620 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.119938 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.120021 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.121347 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:33.121349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:33.121351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:33.121362 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:33.121379 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:33.121386 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:33.121442 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.121654 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.121704 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:37:33.122920 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.114286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:43711 (local address 127.27.46.194:37635)
0925 04:37:33.114923 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.114963 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.116238 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.116716 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.116722 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.116817 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:33.117287 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.117298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.118344 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.118348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.119497 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.119508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.120091 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.120095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.120315 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.121051 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:33.121055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:33.121113 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:33.121198 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:33.121466 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:33.121478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:33.121486 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.121641 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.121966 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260925 04:37:33.124162 15862 heartbeater.cc:347] Connected to a master server at 127.27.46.254:43711
I20260925 04:37:33.124423 15862 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:33.124953 15862 heartbeater.cc:511] Master 127.27.46.254:43711 requested a full tablet report, sending...
I20260925 04:37:33.126137 15536 ts_manager.cc:194] Registered new tserver with Master: f8915fbf281f4c73b6895132871edeaa (127.27.46.194:41125)
I20260925 04:37:33.126726 15536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.194:37635
W20260925 04:37:33.235409 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:33.235683 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:33.235760 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:33.239964 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:33.240031 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:33.240131 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.46.195
I20260925 04:37:33.244978 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.46.195
--webserver_port=0
--tserver_master_addrs=127.27.46.254:43711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.46.195
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:33.246206 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:33.247699 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:33.257480 15871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:33.257480 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:33.257629 15865 server_base.cc:1034] running on GCE node
W20260925 04:37:33.257905 15873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:33.258344 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:33.259021 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:33.260181 15865 hybrid_clock.cc:648] HybridClock initialized: now 1790311053260147 us; error 45 us; skew 500 ppm
I20260925 04:37:33.266870 15865 webserver.cc:460] Webserver started at http://127.27.46.195:43111/ using document root <none> and password file <none>
I20260925 04:37:33.267503 15865 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:33.267571 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:33.267772 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:33.270054 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data/instance:
uuid: "cb96094103974f15a65f3f200ef89cdf"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "22f0fadcc3ccb72b873556378cac7278"
server_key_iv: "bd2c43e708d886621a44226c9db0e58b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.270603 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance:
uuid: "cb96094103974f15a65f3f200ef89cdf"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "22f0fadcc3ccb72b873556378cac7278"
server_key_iv: "bd2c43e708d886621a44226c9db0e58b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.275018 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:33.277727 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.279264 15865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:33.279376 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal
uuid: "cb96094103974f15a65f3f200ef89cdf"
format_stamp: "Formatted at 2026-09-25 04:37:33 on dist-test-slave-r8r3"
server_key: "22f0fadcc3ccb72b873556378cac7278"
server_key_iv: "bd2c43e708d886621a44226c9db0e58b"
server_key_version: "encryptionkey@0"
I20260925 04:37:33.279507 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:33.301604 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:33.302469 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:33.302649 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:33.303350 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:37:33.304491 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:37:33.304549 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.304610 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:37:33.304627 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:33.315202 15865 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.195:45733
I20260925 04:37:33.315238 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.195:45733 every 8 connection(s)
I20260925 04:37:33.316123 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/data/info.pb
I20260925 04:37:33.319916 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 15865
I20260925 04:37:33.320060 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal/instance
I20260925 04:37:33.320365 27835 external_mini_cluster.cc:1428] Setting key 08dad0f6e9e69d01ad1f7c1da6865852
I20260925 04:37:33.326937 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.318260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.195:47949 (local address 127.27.46.254:43711)
0925 04:37:33.318426 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.318431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.319697 (+ 1266us) server_negotiation.cc:407] Connection header received
0925 04:37:33.321771 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.321777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.321828 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.321924 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:33.323008 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.323578 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.324764 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.324997 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.325110 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.326357 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:33.326373 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:33.326376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:33.326397 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:33.326415 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:33.326423 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:33.326519 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.326691 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.326786 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:37:33.327807 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.318683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:43711 (local address 127.27.46.195:47949)
0925 04:37:33.319495 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.319555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.321479 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.322107 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.322113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.322195 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:37:33.322871 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.322894 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.323712 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.323715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.324614 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.324624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.325156 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.325160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.325344 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.326057 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:37:33.326058 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:37:33.326100 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:37:33.326166 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:37:33.326521 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:37:33.326533 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:37:33.326541 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.326691 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.326975 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":132,"threads_started":1}
I20260925 04:37:33.328985 15994 heartbeater.cc:347] Connected to a master server at 127.27.46.254:43711
I20260925 04:37:33.329247 15994 heartbeater.cc:464] Registering TS with master...
I20260925 04:37:33.329950 15994 heartbeater.cc:511] Master 127.27.46.254:43711 requested a full tablet report, sending...
I20260925 04:37:33.331190 15536 ts_manager.cc:194] Registered new tserver with Master: cb96094103974f15a65f3f200ef89cdf (127.27.46.195:45733)
I20260925 04:37:33.331992 15536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.46.195:47949
I20260925 04:37:33.334527 27835 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:37:33.340986 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.336525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.27.46.254:43711)
0925 04:37:33.336671 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.336676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.336926 (+ 250us) server_negotiation.cc:407] Connection header received
0925 04:37:33.337056 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.337058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.337097 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.337161 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:33.337555 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.338052 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.339383 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.339578 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.339652 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.340539 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:33.340542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:33.340544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:33.340555 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:33.340570 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:33.340577 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:33.340632 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.340778 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.340825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:37:33.344115 15536 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:37:33.347290 15536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:37:33.363444 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.356444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:38915 (local address 127.0.0.1:38602)
0925 04:37:33.356783 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.356830 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.356952 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.358307 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.358313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.358336 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:33.358486 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.358493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.359984 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.359988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.361525 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.361537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.362780 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.362785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.362977 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.363028 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.363132 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.363252 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":96,"thread_start_us":84,"threads_started":1}
I20260925 04:37:33.363561 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.357397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.27.46.193:38915)
0925 04:37:33.357779 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.357793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.357821 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:37:33.357904 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.357912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.358133 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.358319 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:33.358611 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.359852 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.361671 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.362605 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.362965 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.363172 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.363229 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.363364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":164,"threads_started":1}
I20260925 04:37:33.364359 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.356743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:45733 (local address 127.0.0.1:53414)
0925 04:37:33.357639 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.357651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.357749 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.358581 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.358584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.358602 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:33.358702 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.358708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.360198 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.360201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.361411 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.361421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.362773 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.362777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.364013 (+ 1236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.364038 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.364111 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.364176 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":120,"thread_start_us":89,"threads_started":1}
I20260925 04:37:33.364935 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.356499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.194:41125 (local address 127.0.0.1:39770)
0925 04:37:33.357317 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.357332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.357432 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.358870 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.358872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.358891 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:33.358967 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.358973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.360434 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.360438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.362353 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.362366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.363649 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.363656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.364625 (+ 969us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.364653 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.364721 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.364812 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":604,"thread_start_us":126,"threads_started":1}
I20260925 04:37:33.365127 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.357258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.27.46.194:41125)
0925 04:37:33.357931 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.357943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.357966 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:37:33.358521 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.358536 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.358718 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.359108 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:33.359239 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.360316 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.362505 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.363491 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.364014 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.364202 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.364841 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.364945 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":96,"threads_started":1}
I20260925 04:37:33.365813 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.357406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.27.46.195:45733)
0925 04:37:33.357999 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.358010 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.358027 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:37:33.358090 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.358095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.358281 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.358426 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:33.358856 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.360068 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.361573 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.362393 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.362492 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.362696 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.365572 (+ 2876us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.365681 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":165,"threads_started":1}
I20260925 04:37:33.367534 15662 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61be6b847e459b8a5250ffaac5096f (DEFAULT_TABLE table=test-table [id=7eb0486976a649b48200f26d4545ad43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:33.367816 15795 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61be6b847e459b8a5250ffaac5096f (DEFAULT_TABLE table=test-table [id=7eb0486976a649b48200f26d4545ad43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:33.368744 15662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61be6b847e459b8a5250ffaac5096f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:33.368463 15924 tablet_service.cc:1467] Processing CreateTablet for tablet 9c61be6b847e459b8a5250ffaac5096f (DEFAULT_TABLE table=test-table [id=7eb0486976a649b48200f26d4545ad43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:37:33.368867 15795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61be6b847e459b8a5250ffaac5096f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:33.369401 15924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c61be6b847e459b8a5250ffaac5096f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:33.377437 16013 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Bootstrap starting.
I20260925 04:37:33.378077 16014 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Bootstrap starting.
I20260925 04:37:33.379168 16015 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Bootstrap starting.
I20260925 04:37:33.380506 16013 tablet_bootstrap.cc:654] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:33.381448 16013 log.cc:824] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:33.382715 16015 tablet_bootstrap.cc:654] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:33.382885 16014 tablet_bootstrap.cc:654] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:33.383723 16014 log.cc:824] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:33.383795 16015 log.cc:824] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:33.383971 16013 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: No bootstrap required, opened a new log
I20260925 04:37:33.384181 16013 ts_tablet_manager.cc:1397] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:37:33.385869 16015 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: No bootstrap required, opened a new log
I20260925 04:37:33.386040 16015 ts_tablet_manager.cc:1397] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 04:37:33.387066 16014 tablet_bootstrap.cc:492] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: No bootstrap required, opened a new log
I20260925 04:37:33.387225 16014 ts_tablet_manager.cc:1397] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 04:37:33.389364 16013 raft_consensus.cc:348] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.390062 16013 raft_consensus.cc:374] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:33.390175 16013 raft_consensus.cc:729] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb96094103974f15a65f3f200ef89cdf, State: Initialized, Role: FOLLOWER
I20260925 04:37:33.390362 16015 raft_consensus.cc:348] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.390610 16015 raft_consensus.cc:374] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:33.390672 16015 raft_consensus.cc:729] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8915fbf281f4c73b6895132871edeaa, State: Initialized, Role: FOLLOWER
I20260925 04:37:33.390901 16013 consensus_queue.cc:260] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.391124 16015 consensus_queue.cc:260] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.391743 15994 heartbeater.cc:503] Master 127.27.46.254:43711 was elected leader, sending a full tablet report...
I20260925 04:37:33.392118 16015 ts_tablet_manager.cc:1428] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 04:37:33.392241 16013 ts_tablet_manager.cc:1428] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 04:37:33.392462 15862 heartbeater.cc:503] Master 127.27.46.254:43711 was elected leader, sending a full tablet report...
I20260925 04:37:33.391412 16014 raft_consensus.cc:348] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.392639 16014 raft_consensus.cc:374] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:33.392700 16014 raft_consensus.cc:729] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b16fa223ca46f08a86065ee0b95fa6, State: Initialized, Role: FOLLOWER
I20260925 04:37:33.393141 16014 consensus_queue.cc:260] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.394073 16014 ts_tablet_manager.cc:1428] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 04:37:33.394513 15730 heartbeater.cc:503] Master 127.27.46.254:43711 was elected leader, sending a full tablet report...
W20260925 04:37:33.421599 15731 tablet.cc:2369] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:33.533820 16020 raft_consensus.cc:482] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:37:33.534083 16020 raft_consensus.cc:504] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.535459 16020 leader_election.cc:288] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b16fa223ca46f08a86065ee0b95fa6 (127.27.46.193:38915), cb96094103974f15a65f3f200ef89cdf (127.27.46.195:45733)
I20260925 04:37:33.541152 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.535925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:55481 (local address 127.27.46.193:38915)
0925 04:37:33.536184 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.536190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.536245 (+ 55us) server_negotiation.cc:407] Connection header received
0925 04:37:33.536391 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.536395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.536447 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.536535 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:33.536880 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.537713 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.539731 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.540391 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.540536 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.540624 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.540907 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.540993 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260925 04:37:33.541152 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.535870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.46.194:50243 (local address 127.27.46.195:45733)
0925 04:37:33.536010 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.536016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.536037 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:37:33.536175 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.536179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.536238 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.536336 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:37:33.536866 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.537686 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.539731 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.540391 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.540536 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.540600 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.540907 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.540993 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:37:33.541181 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.535733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.195:45733 (local address 127.27.46.194:50243)
0925 04:37:33.535926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.535941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.536043 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.536643 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.536645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.536658 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:33.536740 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.536745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.537845 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.537848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.539569 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.539581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.540537 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.540541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.540701 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.540773 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.540854 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.541002 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":326}
I20260925 04:37:33.541190 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.535798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:38915 (local address 127.27.46.194:55481)
0925 04:37:33.536165 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.536180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:37:33.536267 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:37:33.536548 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:37:33.536551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:37:33.536568 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:37:33.536710 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.536718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.537845 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.537848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.539569 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:37:33.539581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.540537 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.540541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:37:33.540698 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.540777 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 04:37:33.540854 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:37:33.541002 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":293,"thread_start_us":128,"threads_started":1}
I20260925 04:37:33.542027 15943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61be6b847e459b8a5250ffaac5096f" candidate_uuid: "f8915fbf281f4c73b6895132871edeaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb96094103974f15a65f3f200ef89cdf" is_pre_election: true
I20260925 04:37:33.542073 15682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61be6b847e459b8a5250ffaac5096f" candidate_uuid: "f8915fbf281f4c73b6895132871edeaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b16fa223ca46f08a86065ee0b95fa6" is_pre_election: true
I20260925 04:37:33.542373 15943 raft_consensus.cc:2454] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8915fbf281f4c73b6895132871edeaa in term 0.
I20260925 04:37:33.542373 15682 raft_consensus.cc:2454] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8915fbf281f4c73b6895132871edeaa in term 0.
I20260925 04:37:33.542937 15749 leader_election.cc:302] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b16fa223ca46f08a86065ee0b95fa6, f8915fbf281f4c73b6895132871edeaa; no voters:
I20260925 04:37:33.543255 16020 raft_consensus.cc:2781] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:37:33.543352 16020 raft_consensus.cc:482] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:37:33.543403 16020 raft_consensus.cc:3037] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:33.544471 16020 raft_consensus.cc:504] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.544885 16020 leader_election.cc:288] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [CANDIDATE]: Term 1 election: Requested vote from peers 73b16fa223ca46f08a86065ee0b95fa6 (127.27.46.193:38915), cb96094103974f15a65f3f200ef89cdf (127.27.46.195:45733)
I20260925 04:37:33.545251 15682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61be6b847e459b8a5250ffaac5096f" candidate_uuid: "f8915fbf281f4c73b6895132871edeaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b16fa223ca46f08a86065ee0b95fa6"
I20260925 04:37:33.545269 15943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c61be6b847e459b8a5250ffaac5096f" candidate_uuid: "f8915fbf281f4c73b6895132871edeaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb96094103974f15a65f3f200ef89cdf"
I20260925 04:37:33.545397 15943 raft_consensus.cc:3037] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:33.545361 15682 raft_consensus.cc:3037] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:33.546412 15943 raft_consensus.cc:2454] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8915fbf281f4c73b6895132871edeaa in term 1.
I20260925 04:37:33.546480 15682 raft_consensus.cc:2454] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8915fbf281f4c73b6895132871edeaa in term 1.
I20260925 04:37:33.546784 15750 leader_election.cc:302] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb96094103974f15a65f3f200ef89cdf, f8915fbf281f4c73b6895132871edeaa; no voters:
I20260925 04:37:33.546998 16020 raft_consensus.cc:2781] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:33.547240 16020 raft_consensus.cc:686] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 1 LEADER]: Becoming Leader. State: Replica: f8915fbf281f4c73b6895132871edeaa, State: Running, Role: LEADER
I20260925 04:37:33.547578 16020 consensus_queue.cc:237] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } }
I20260925 04:37:33.550747 15535 catalog_manager.cc:5495] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa reported cstate change: term changed from 0 to 1, leader changed from <none> to f8915fbf281f4c73b6895132871edeaa (127.27.46.194). New cstate: current_term: 1 leader_uuid: "f8915fbf281f4c73b6895132871edeaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8915fbf281f4c73b6895132871edeaa" member_type: VOTER last_known_addr { host: "127.27.46.194" port: 41125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 } health_report { overall_health: UNKNOWN } } }
I20260925 04:37:33.562702 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.559281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.27.46.194:41125)
0925 04:37:33.559404 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:33.559408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:33.559443 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:37:33.559512 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:33.559514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:33.559565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:33.559643 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:37:33.559856 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.560346 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.561289 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:33.561551 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:33.561634 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:33.561898 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:37:33.562320 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:37:33.562411 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:33.562521 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 04:37:33.562568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:37:33.567072 15995 tablet.cc:2369] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:37:33.568619 15943 raft_consensus.cc:1261] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 1 FOLLOWER]: Refusing update from remote peer f8915fbf281f4c73b6895132871edeaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:33.568747 15682 raft_consensus.cc:1261] T 9c61be6b847e459b8a5250ffaac5096f P 73b16fa223ca46f08a86065ee0b95fa6 [term 1 FOLLOWER]: Refusing update from remote peer f8915fbf281f4c73b6895132871edeaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:37:33.569423 16023 consensus_queue.cc:1035] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb96094103974f15a65f3f200ef89cdf" member_type: VOTER last_known_addr { host: "127.27.46.195" port: 45733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:33.569657 16020 consensus_queue.cc:1035] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b16fa223ca46f08a86065ee0b95fa6" member_type: VOTER last_known_addr { host: "127.27.46.193" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:37:33.577252 16027 mvcc.cc:205] Tried to move back new op lower bound from 7333114075409485824 to 7333114075333681152. Current Snapshot: MvccSnapshot[applied={T|T < 7333114075409485824}]
I20260925 04:37:33.578538 16028 mvcc.cc:205] Tried to move back new op lower bound from 7333114075409485824 to 7333114075333681152. Current Snapshot: MvccSnapshot[applied={T|T < 7333114075409485824}]
I20260925 04:37:33.585922 15536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:33.586122 15536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:37:33.588581 15536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17867fd422494d49b6951f83d0b0e2d6: Sending DeleteTablet for 3 replicas of tablet 9c61be6b847e459b8a5250ffaac5096f
I20260925 04:37:33.589346 15795 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61be6b847e459b8a5250ffaac5096f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:33 UTC) from {username='slave'} at 127.0.0.1:39770
I20260925 04:37:33.589447 15662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61be6b847e459b8a5250ffaac5096f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:33 UTC) from {username='slave'} at 127.0.0.1:38602
I20260925 04:37:33.590001 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15599
I20260925 04:37:33.590267 15924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c61be6b847e459b8a5250ffaac5096f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:37:33 UTC) from {username='slave'} at 127.0.0.1:53414
I20260925 04:37:33.590700 16036 tablet_replica.cc:330] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: stopping tablet replica
I20260925 04:37:33.591017 16036 raft_consensus.cc:2229] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:37:33.591375 16036 raft_consensus.cc:2258] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:37:33.592895 16036 ts_tablet_manager.cc:1905] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:37:33.595749 16036 ts_tablet_manager.cc:1918] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:33.595852 16036 log.cc:1196] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-1/wal/wals/9c61be6b847e459b8a5250ffaac5096f
I20260925 04:37:33.596163 16036 ts_tablet_manager.cc:1939] T 9c61be6b847e459b8a5250ffaac5096f P f8915fbf281f4c73b6895132871edeaa: Deleting consensus metadata
I20260925 04:37:33.597029 16038 tablet_replica.cc:330] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: stopping tablet replica
I20260925 04:37:33.597378 15520 catalog_manager.cc:4841] TS f8915fbf281f4c73b6895132871edeaa (127.27.46.194:41125): tablet 9c61be6b847e459b8a5250ffaac5096f (table test-table [id=7eb0486976a649b48200f26d4545ad43]) successfully deleted
I20260925 04:37:33.597445 16038 raft_consensus.cc:2229] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:37:33.597951 16038 raft_consensus.cc:2258] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:37:33.598294 15521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.46.193:38915 (error 108)
I20260925 04:37:33.599355 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:33.599049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.193:38915 (local address 127.0.0.1:38618)
0925 04:37:33.599194 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 04:37:33.599268 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.193:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:37:33.599406 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15733
I20260925 04:37:33.599687 16038 ts_tablet_manager.cc:1905] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 04:37:33.602910 15521 catalog_manager.cc:4584] TS 73b16fa223ca46f08a86065ee0b95fa6 (127.27.46.193:38915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c61be6b847e459b8a5250ffaac5096f: Network error: Client connection negotiation failed: client connection to 127.27.46.193:38915: connect: Connection refused (error 111)
I20260925 04:37:33.603117 16038 ts_tablet_manager.cc:1918] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:37:33.603231 16038 log.cc:1196] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Deleting WAL directory at /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790310801918616-27835-0/minicluster-data/ts-2/wal/wals/9c61be6b847e459b8a5250ffaac5096f
I20260925 04:37:33.603572 16038 ts_tablet_manager.cc:1939] T 9c61be6b847e459b8a5250ffaac5096f P cb96094103974f15a65f3f200ef89cdf: Deleting consensus metadata
I20260925 04:37:33.604931 15522 catalog_manager.cc:4841] TS cb96094103974f15a65f3f200ef89cdf (127.27.46.195:45733): tablet 9c61be6b847e459b8a5250ffaac5096f (table test-table [id=7eb0486976a649b48200f26d4545ad43]) successfully deleted
I20260925 04:37:33.608415 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15865
I20260925 04:37:33.615315 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 15505
2026-09-25T04:37:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1165 ms)
[----------] 6 tests from EncryptionPolicyTest (10538 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:33 dist-test-slave-r8r3 krb5kdc[16043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:33 dist-test-slave-r8r3 krb5kdc[16043](info): set up 2 sockets
Sep 25 04:37:33 dist-test-slave-r8r3 krb5kdc[16043](info): commencing operation
krb5kdc: starting...
W20260925 04:37:34.666167 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:34 dist-test-slave-r8r3 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:34.792016 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:34.907232 16056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:34.907577 16056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:34.907682 16056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:34.911891 16056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:34.911998 16056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:34.912039 16056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:34.912101 16056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:34.912138 16056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:34.917766 16056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:34.918939 16056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:34.920003 16056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:34.928571 16061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:34.928619 16062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:34.928633 16064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:34.929033 16056 server_base.cc:1034] running on GCE node
I20260925 04:37:34.929728 16056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:34.930850 16056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:34.932041 16056 hybrid_clock.cc:648] HybridClock initialized: now 1790311054932039 us; error 40 us; skew 500 ppm
Sep 25 04:37:34 dist-test-slave-r8r3 krb5kdc[16043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:34.934942 16056 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:34.935887 16056 webserver.cc:460] Webserver started at http://127.0.0.1:46709/ using document root <none> and password file <none>
I20260925 04:37:34.936484 16056 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:34.936579 16056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:34.936839 16056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:34.939127 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "135ae1703e0a4b94a743be743cc71b4c"
format_stamp: "Formatted at 2026-09-25 04:37:34 on dist-test-slave-r8r3"
server_key: "99e6b66e41fd555bf3711f7ea50e550d"
server_key_iv: "d58928641c57c13e41086e08f3bb611b"
server_key_version: "encryptionkey@0"
I20260925 04:37:34.939728 16056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "135ae1703e0a4b94a743be743cc71b4c"
format_stamp: "Formatted at 2026-09-25 04:37:34 on dist-test-slave-r8r3"
server_key: "99e6b66e41fd555bf3711f7ea50e550d"
server_key_iv: "d58928641c57c13e41086e08f3bb611b"
server_key_version: "encryptionkey@0"
I20260925 04:37:34.943699 16056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:34.946389 16071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:34.947738 16056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:34.947897 16056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "135ae1703e0a4b94a743be743cc71b4c"
format_stamp: "Formatted at 2026-09-25 04:37:34 on dist-test-slave-r8r3"
server_key: "99e6b66e41fd555bf3711f7ea50e550d"
server_key_iv: "d58928641c57c13e41086e08f3bb611b"
server_key_version: "encryptionkey@0"
I20260925 04:37:34.948020 16056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:34.963418 16056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:34.967041 16056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:34.967278 16056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:34.974650 16056 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41205
I20260925 04:37:34.974664 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41205 every 8 connection(s)
I20260925 04:37:34.975647 16056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:34.978847 16124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:34.979310 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16056
I20260925 04:37:34.979480 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:34.979784 27835 external_mini_cluster.cc:1428] Setting key b3cc9c446bd77f71d95b35548f247f27
I20260925 04:37:34.986755 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c: Bootstrap starting.
I20260925 04:37:34.989583 16124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c: Neither blocks nor log segments found. Creating new log.
Sep 25 04:37:34 dist-test-slave-r8r3 krb5kdc[16043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:37:34.990592 16124 log.cc:824] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:34.993189 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c: No bootstrap required, opened a new log
I20260925 04:37:34.995966 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:34.981314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48855 (local address 127.0.0.1:41205)
0925 04:37:34.981787 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:34.981808 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:34.981855 (+ 47us) server_negotiation.cc:407] Connection header received
0925 04:37:34.982623 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:34.982647 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:34.983154 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:34.983557 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:34.984259 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:34.985837 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:34.986840 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:34.987570 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:34.987753 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:34.990721 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:34.990730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:34.990749 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:34.990791 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:34.992849 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:34.993481 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:34.993486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:34.993531 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:34.993656 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:34.994032 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:34.994042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:34.994044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:34.994427 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:34.994650 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:34.995006 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:37:34.995235 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":169,"threads_started":1}
I20260925 04:37:34.997104 16124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260925 04:37:34.997380 16124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:34.997447 16124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135ae1703e0a4b94a743be743cc71b4c, State: Initialized, Role: FOLLOWER
I20260925 04:37:34.997968 16124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260925 04:37:34.998143 16124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:34.998209 16124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:34.998332 16124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:34.999142 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:34.999550 16124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260925 04:37:34.999899 16124 leader_election.cc:302] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135ae1703e0a4b94a743be743cc71b4c; no voters:
I20260925 04:37:35.000227 16124 leader_election.cc:288] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:35.000310 16129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:35.000471 16129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [term 1 LEADER]: Becoming Leader. State: Replica: 135ae1703e0a4b94a743be743cc71b4c, State: Running, Role: LEADER
I20260925 04:37:35.000758 16129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } }
I20260925 04:37:35.001550 16124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:35.002022 16134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ae1703e0a4b94a743be743cc71b4c. Latest consensus state: current_term: 1 leader_uuid: "135ae1703e0a4b94a743be743cc71b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } } }
I20260925 04:37:35.002177 16134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:35.002434 16135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135ae1703e0a4b94a743be743cc71b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ae1703e0a4b94a743be743cc71b4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41205 } } }
I20260925 04:37:35.002540 16135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:35.002701 16139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:35.006177 16139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:35.010859 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:35.001272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46265 (local address 127.0.0.1:41205)
0925 04:37:35.001404 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:35.001408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:35.001691 (+ 283us) server_negotiation.cc:407] Connection header received
0925 04:37:35.001890 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:35.001893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:35.001941 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:35.001993 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:35.003033 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:35.003788 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:35.004800 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:35.005179 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:35.005278 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:35.007252 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:35.007256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:35.007258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:35.007292 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:35.008790 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:35.009366 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:35.009372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:35.009375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:35.009486 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:35.009904 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:35.009907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:35.009909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:35.010133 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:35.010293 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:35.010595 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 04:37:35.010697 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260925 04:37:35.013175 16139 catalog_manager.cc:1347] Generated new cluster ID: 69d1e04a5445461fb6b2059f69d80c5f
I20260925 04:37:35.013253 16139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:35.029143 16139 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:35.030220 16139 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:35.040277 16139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135ae1703e0a4b94a743be743cc71b4c: Generated new TSK 0
I20260925 04:37:35.041086 16139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:35.047113 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16056
W20260925 04:37:35.053117 16040 connection.cc:535] client connection to 127.0.0.1:41205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41205): Connection reset by peer (error 104)
2026-09-25T04:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:35 dist-test-slave-r8r3 krb5kdc[16157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:35 dist-test-slave-r8r3 krb5kdc[16157](info): set up 2 sockets
Sep 25 04:37:35 dist-test-slave-r8r3 krb5kdc[16157](info): commencing operation
krb5kdc: starting...
W20260925 04:37:36.093783 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:36 dist-test-slave-r8r3 krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:36.214707 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:36.331912 16170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:36.332199 16170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:36.332269 16170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:36.335978 16170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:36.336063 16170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:36.336098 16170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:36.336153 16170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:36.336189 16170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:36.341382 16170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:36.342525 16170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:36.343611 16170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:36.352334 16175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:36.352362 16176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:36.352334 16178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:36.352424 16170 server_base.cc:1034] running on GCE node
I20260925 04:37:36.353173 16170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:36.354331 16170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:36.355500 16170 hybrid_clock.cc:648] HybridClock initialized: now 1790311056355482 us; error 30 us; skew 500 ppm
Sep 25 04:37:36 dist-test-slave-r8r3 krb5kdc[16157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:36.358274 16170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:36.359242 16170 webserver.cc:460] Webserver started at http://127.0.0.1:41685/ using document root <none> and password file <none>
I20260925 04:37:36.359845 16170 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:36.359907 16170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:36.360158 16170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:36.362426 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "09205c91a2234411a15cc761c2f04f14"
format_stamp: "Formatted at 2026-09-25 04:37:36 on dist-test-slave-r8r3"
server_key: "7d0d484249b6b4142efe0ad496cbdacb"
server_key_iv: "a0ec4f7a2ab5d63d926a91bf5f79e454"
server_key_version: "encryptionkey@0"
I20260925 04:37:36.363055 16170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "09205c91a2234411a15cc761c2f04f14"
format_stamp: "Formatted at 2026-09-25 04:37:36 on dist-test-slave-r8r3"
server_key: "7d0d484249b6b4142efe0ad496cbdacb"
server_key_iv: "a0ec4f7a2ab5d63d926a91bf5f79e454"
server_key_version: "encryptionkey@0"
I20260925 04:37:36.367029 16170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:36.369791 16185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:36.371088 16170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:36.371245 16170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "09205c91a2234411a15cc761c2f04f14"
format_stamp: "Formatted at 2026-09-25 04:37:36 on dist-test-slave-r8r3"
server_key: "7d0d484249b6b4142efe0ad496cbdacb"
server_key_iv: "a0ec4f7a2ab5d63d926a91bf5f79e454"
server_key_version: "encryptionkey@0"
I20260925 04:37:36.371374 16170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:36.390692 16170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:36.393805 16170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:36.394023 16170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:36.401207 16170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33243
I20260925 04:37:36.401254 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33243 every 8 connection(s)
I20260925 04:37:36.402228 16170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:36.405040 16238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:36.410774 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14: Bootstrap starting.
I20260925 04:37:36.411430 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16170
I20260925 04:37:36.411594 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:36.411898 27835 external_mini_cluster.cc:1428] Setting key 57276268639c9e3e04d420febce1f0e1
I20260925 04:37:36.413921 16238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:36.414731 16238 log.cc:824] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:36.416887 16238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14: No bootstrap required, opened a new log
I20260925 04:37:36.421389 16238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260925 04:37:36.421705 16238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:36.421800 16238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09205c91a2234411a15cc761c2f04f14, State: Initialized, Role: FOLLOWER
I20260925 04:37:36.422058 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:36.413712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35165 (local address 127.0.0.1:33243)
0925 04:37:36.414175 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:36.414191 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:36.414228 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:37:36.414988 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:36.415007 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:36.415526 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:36.415843 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:36.416185 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:36.417238 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:36.418450 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:36.418935 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:36.419060 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:36.420030 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:36.420036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:36.420050 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:36.420471 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:36.420528 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:36.420551 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:36.420654 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:36.420877 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:37:36.421056 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":157,"threads_started":1}
I20260925 04:37:36.422334 16238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260925 04:37:36.422502 16238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:36.422570 16238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:36.422667 16238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:36.423758 16238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260925 04:37:36.424144 16238 leader_election.cc:302] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09205c91a2234411a15cc761c2f04f14; no voters:
I20260925 04:37:36.424503 16238 leader_election.cc:288] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:36.424625 16243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:36.424868 16243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [term 1 LEADER]: Becoming Leader. State: Replica: 09205c91a2234411a15cc761c2f04f14, State: Running, Role: LEADER
I20260925 04:37:36.425189 16243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260925 04:37:36.425326 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:36.425915 16238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:36.426838 16245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09205c91a2234411a15cc761c2f04f14. Latest consensus state: current_term: 1 leader_uuid: "09205c91a2234411a15cc761c2f04f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } } }
I20260925 04:37:36.426959 16245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:36.427314 16244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09205c91a2234411a15cc761c2f04f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09205c91a2234411a15cc761c2f04f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } } }
I20260925 04:37:36.427403 16244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:36.428301 16256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:36.431641 16256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:36.433346 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:36.428507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45385 (local address 127.0.0.1:33243)
0925 04:37:36.428604 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:36.428609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:36.428948 (+ 339us) server_negotiation.cc:407] Connection header received
0925 04:37:36.429079 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:36.429082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:36.429133 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:36.429221 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:36.429518 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:36.430280 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:36.431460 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:36.431806 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:36.431912 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:36.432843 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:36.432847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:36.432849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:36.432864 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:36.432891 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:36.432903 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:36.432994 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:36.433165 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:37:36.433202 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 04:37:36.438019 16256 catalog_manager.cc:1347] Generated new cluster ID: 6599f22ecb23461e8532e5d49eb5100d
I20260925 04:37:36.438103 16256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:36.470118 16256 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:36.471076 16256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:36.475656 16256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09205c91a2234411a15cc761c2f04f14: Generated new TSK 0
I20260925 04:37:36.476331 16256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:36.486274 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16170
W20260925 04:37:36.492568 16155 connection.cc:535] client connection to 127.0.0.1:33243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33243): Connection reset by peer (error 104)
2026-09-25T04:37:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:36 dist-test-slave-r8r3 krb5kdc[16271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:36 dist-test-slave-r8r3 krb5kdc[16271](info): set up 2 sockets
Sep 25 04:37:36 dist-test-slave-r8r3 krb5kdc[16271](info): commencing operation
krb5kdc: starting...
W20260925 04:37:37.537647 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:37 dist-test-slave-r8r3 krb5kdc[16271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:37.664636 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:37.776933 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:37.777235 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:37.777285 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:37.781265 16284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:37.781325 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:37.781344 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:37.781361 16284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:37.781376 16284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:37.786757 16284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:37.787889 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:37.788913 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:37.797791 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:37.797868 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:37.797869 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:37.797887 16284 server_base.cc:1034] running on GCE node
I20260925 04:37:37.798637 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:37.799825 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:37.801024 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790311057801010 us; error 26 us; skew 500 ppm
Sep 25 04:37:37 dist-test-slave-r8r3 krb5kdc[16271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:37.803768 16284 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:37.804669 16284 webserver.cc:460] Webserver started at http://127.0.0.1:44527/ using document root <none> and password file <none>
I20260925 04:37:37.805240 16284 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:37.805349 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:37.805652 16284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:37.807778 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "3f7420a776f64ff0accd297b02eeda5e"
format_stamp: "Formatted at 2026-09-25 04:37:37 on dist-test-slave-r8r3"
server_key: "cf7916066880a62ce1ca4fb13d92edfb"
server_key_iv: "c2d1c02b7bbcd73b3c0401cf5528a732"
server_key_version: "encryptionkey@0"
I20260925 04:37:37.808377 16284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "3f7420a776f64ff0accd297b02eeda5e"
format_stamp: "Formatted at 2026-09-25 04:37:37 on dist-test-slave-r8r3"
server_key: "cf7916066880a62ce1ca4fb13d92edfb"
server_key_iv: "c2d1c02b7bbcd73b3c0401cf5528a732"
server_key_version: "encryptionkey@0"
I20260925 04:37:37.812167 16284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:37:37.814672 16299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:37.815954 16284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:37:37.816100 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "3f7420a776f64ff0accd297b02eeda5e"
format_stamp: "Formatted at 2026-09-25 04:37:37 on dist-test-slave-r8r3"
server_key: "cf7916066880a62ce1ca4fb13d92edfb"
server_key_iv: "c2d1c02b7bbcd73b3c0401cf5528a732"
server_key_version: "encryptionkey@0"
I20260925 04:37:37.816227 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:37.820960 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:37.823952 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:37.824159 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:37.831432 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37423
I20260925 04:37:37.831442 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37423 every 8 connection(s)
I20260925 04:37:37.832432 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:37.835306 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:37.840900 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e: Bootstrap starting.
I20260925 04:37:37.841310 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16284
I20260925 04:37:37.841460 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:37.841770 27835 external_mini_cluster.cc:1428] Setting key e5533c2c42aa8c06cbe0659b17b8c7d1
I20260925 04:37:37.843739 16352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:37.844491 16352 log.cc:824] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:37.846851 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e: No bootstrap required, opened a new log
I20260925 04:37:37.847276 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:37.843099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55585 (local address 127.0.0.1:37423)
0925 04:37:37.843612 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:37.843627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:37.843663 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:37:37.844362 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:37.844379 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:37.844727 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:37.845105 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:37.845936 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:37.845940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:37.845957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:37.845966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:37.846016 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:37.846035 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:37.846126 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:37.846281 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 04:37:37.846468 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":172,"threads_started":1}
I20260925 04:37:37.850302 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:37.851173 16352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260925 04:37:37.851444 16352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:37.851562 16352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7420a776f64ff0accd297b02eeda5e, State: Initialized, Role: FOLLOWER
I20260925 04:37:37.852037 16352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260925 04:37:37.852197 16352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:37.852278 16352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:37.852423 16352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:37.853892 16352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260925 04:37:37.854251 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:37.852343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37777 (local address 127.0.0.1:37423)
0925 04:37:37.852470 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:37.852476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:37.852780 (+ 304us) server_negotiation.cc:407] Connection header received
0925 04:37:37.852906 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:37.852908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:37.852942 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:37.853008 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:37.853754 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:37.853757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:37.853759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:37.853769 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:37.853795 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:37.853806 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:37.853877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:37.854044 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:37:37.854091 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 04:37:37.854301 16352 leader_election.cc:302] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f7420a776f64ff0accd297b02eeda5e; no voters:
I20260925 04:37:37.854681 16352 leader_election.cc:288] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:37.854776 16362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:37.854984 16362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [term 1 LEADER]: Becoming Leader. State: Replica: 3f7420a776f64ff0accd297b02eeda5e, State: Running, Role: LEADER
I20260925 04:37:37.855378 16362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260925 04:37:37.855978 16352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:37.857024 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f7420a776f64ff0accd297b02eeda5e. Latest consensus state: current_term: 1 leader_uuid: "3f7420a776f64ff0accd297b02eeda5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } } }
I20260925 04:37:37.857167 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:37.857518 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f7420a776f64ff0accd297b02eeda5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7420a776f64ff0accd297b02eeda5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } } }
I20260925 04:37:37.857657 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:37.857581 16371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:37.861099 16371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:37.867224 16371 catalog_manager.cc:1347] Generated new cluster ID: cb5d8275291c409b93bfb7b6c5bbf4e4
I20260925 04:37:37.867328 16371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:37.913306 16371 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:37.913724 16371 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:37.925200 16371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f7420a776f64ff0accd297b02eeda5e: Generated new TSK 0
I20260925 04:37:37.926055 16371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:37.934995 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16284
2026-09-25T04:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:37 dist-test-slave-r8r3 krb5kdc[16383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:37 dist-test-slave-r8r3 krb5kdc[16383](info): set up 2 sockets
Sep 25 04:37:37 dist-test-slave-r8r3 krb5kdc[16383](info): commencing operation
krb5kdc: starting...
W20260925 04:37:38.985981 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:39.112171 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:39.223014 16398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:39.223305 16398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:39.223387 16398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:39.227296 16398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:39.227362 16398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:39.227382 16398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:39.227399 16398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:39.227414 16398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:39.232885 16398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:39.233997 16398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:39.235001 16398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:39.243883 16404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:39.243954 16398 server_base.cc:1034] running on GCE node
W20260925 04:37:39.243876 16403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:39.243883 16406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:39.244822 16398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:39.246078 16398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:39.247259 16398 hybrid_clock.cc:648] HybridClock initialized: now 1790311059247228 us; error 42 us; skew 500 ppm
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:39.254277 16398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:39.255288 16398 webserver.cc:460] Webserver started at http://127.0.0.1:35691/ using document root <none> and password file <none>
I20260925 04:37:39.255892 16398 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:39.255981 16398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:39.256215 16398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:39.258445 16398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "9aa3a7dff84245e68629a88274abcb0b"
format_stamp: "Formatted at 2026-09-25 04:37:39 on dist-test-slave-r8r3"
server_key: "b7c5f91118a2c5572989cbb089825e37"
server_key_iv: "6762c06e464e784b626f142151648053"
server_key_version: "encryptionkey@0"
I20260925 04:37:39.259027 16398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "9aa3a7dff84245e68629a88274abcb0b"
format_stamp: "Formatted at 2026-09-25 04:37:39 on dist-test-slave-r8r3"
server_key: "b7c5f91118a2c5572989cbb089825e37"
server_key_iv: "6762c06e464e784b626f142151648053"
server_key_version: "encryptionkey@0"
I20260925 04:37:39.262977 16398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:39.265559 16413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:39.266813 16398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:37:39.266958 16398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "9aa3a7dff84245e68629a88274abcb0b"
format_stamp: "Formatted at 2026-09-25 04:37:39 on dist-test-slave-r8r3"
server_key: "b7c5f91118a2c5572989cbb089825e37"
server_key_iv: "6762c06e464e784b626f142151648053"
server_key_version: "encryptionkey@0"
I20260925 04:37:39.267079 16398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:39.291096 16398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:39.294211 16398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:39.294431 16398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:39.301959 16398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41407
I20260925 04:37:39.301983 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41407 every 8 connection(s)
I20260925 04:37:39.302956 16398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:39.305744 16468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:39.309005 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16398
I20260925 04:37:39.309115 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:39.309422 27835 external_mini_cluster.cc:1428] Setting key 9defd33b3288ef7d03a3e19aa3a8741d
I20260925 04:37:39.312039 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b: Bootstrap starting.
I20260925 04:37:39.314818 16468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:39.315709 16468 log.cc:824] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:39.318004 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b: No bootstrap required, opened a new log
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:37:39.322162 16468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260925 04:37:39.322476 16468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:39.322574 16468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa3a7dff84245e68629a88274abcb0b, State: Initialized, Role: FOLLOWER
I20260925 04:37:39.323130 16468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260925 04:37:39.323266 16468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:39.323374 16468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:39.323499 16468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:39.324640 16468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260925 04:37:39.325011 16468 leader_election.cc:302] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aa3a7dff84245e68629a88274abcb0b; no voters:
I20260925 04:37:39.325176 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:39.310892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54807 (local address 127.0.0.1:41407)
0925 04:37:39.311340 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:39.311359 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:39.311402 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:37:39.312235 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:39.312259 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:39.312676 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:39.313020 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:39.313729 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:39.315272 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:39.316322 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:39.316964 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:39.317118 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:39.320046 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:39.320055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:39.320074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:39.320105 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:39.321970 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:39.322511 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:39.322518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:39.322524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:39.322691 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:39.322972 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:39.322974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:39.322976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:39.323503 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:39.323705 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:39.323973 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:37:39.324308 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":116,"threads_started":1}
I20260925 04:37:39.325407 16468 leader_election.cc:288] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:39.325543 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:39.325766 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [term 1 LEADER]: Becoming Leader. State: Replica: 9aa3a7dff84245e68629a88274abcb0b, State: Running, Role: LEADER
I20260925 04:37:39.326154 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } }
I20260925 04:37:39.326786 16468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:39.327492 16473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aa3a7dff84245e68629a88274abcb0b. Latest consensus state: current_term: 1 leader_uuid: "9aa3a7dff84245e68629a88274abcb0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } } }
I20260925 04:37:39.327757 16473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:39.328227 16482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:39.328737 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:39.329882 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aa3a7dff84245e68629a88274abcb0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aa3a7dff84245e68629a88274abcb0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41407 } } }
I20260925 04:37:39.330000 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:39.332628 16482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:39.338663 16482 catalog_manager.cc:1347] Generated new cluster ID: 73b6ad4bbd554c31b01a15eea308f085
I20260925 04:37:39.338745 16482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:39.341611 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:39.331464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56793 (local address 127.0.0.1:41407)
0925 04:37:39.331574 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:39.331578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:39.331594 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:37:39.331767 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:39.331770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:39.331862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:39.331966 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:39.332841 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:39.333682 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:39.334897 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:39.335275 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:39.335374 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:39.337197 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:39.337201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:39.337204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:39.337232 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:39.339169 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:39.339691 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:39.339694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:39.339696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:39.339784 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:39.340081 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:39.340084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:39.340086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:39.340329 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:39.340458 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:39.341259 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 04:37:39.341387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:37:39.350174 16482 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:39.351131 16482 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:39.356689 16482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aa3a7dff84245e68629a88274abcb0b: Generated new TSK 0
I20260925 04:37:39.357398 16482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:39.366271 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16398
2026-09-25T04:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16500](info): set up 2 sockets
Sep 25 04:37:39 dist-test-slave-r8r3 krb5kdc[16500](info): commencing operation
krb5kdc: starting...
W20260925 04:37:40.422039 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:40 dist-test-slave-r8r3 krb5kdc[16500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:40.546757 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:40.658308 16514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:40.658584 16514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:40.658627 16514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:40.662870 16514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:40.662935 16514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:40.662953 16514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:40.662971 16514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:40.662985 16514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:40.668387 16514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:40.669533 16514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:40.670615 16514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:40.679504 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:40.679538 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:40.679543 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:40.679596 16514 server_base.cc:1034] running on GCE node
I20260925 04:37:40.680456 16514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:40.681720 16514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:40.682880 16514 hybrid_clock.cc:648] HybridClock initialized: now 1790311060682849 us; error 43 us; skew 500 ppm
Sep 25 04:37:40 dist-test-slave-r8r3 krb5kdc[16500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:40.685593 16514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:40.686484 16514 webserver.cc:460] Webserver started at http://127.0.0.1:45365/ using document root <none> and password file <none>
I20260925 04:37:40.687067 16514 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:40.687161 16514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:40.687399 16514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:40.689566 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "6fa7395a910740e180e7c81d29cb7f8f"
format_stamp: "Formatted at 2026-09-25 04:37:40 on dist-test-slave-r8r3"
server_key: "c550600020477f7f93fc612086ece343"
server_key_iv: "0eb1adf465e26b31407fe76f65c2173b"
server_key_version: "encryptionkey@0"
I20260925 04:37:40.690162 16514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "6fa7395a910740e180e7c81d29cb7f8f"
format_stamp: "Formatted at 2026-09-25 04:37:40 on dist-test-slave-r8r3"
server_key: "c550600020477f7f93fc612086ece343"
server_key_iv: "0eb1adf465e26b31407fe76f65c2173b"
server_key_version: "encryptionkey@0"
I20260925 04:37:40.694022 16514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:40.696643 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:40.697952 16514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:40.698104 16514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "6fa7395a910740e180e7c81d29cb7f8f"
format_stamp: "Formatted at 2026-09-25 04:37:40 on dist-test-slave-r8r3"
server_key: "c550600020477f7f93fc612086ece343"
server_key_iv: "0eb1adf465e26b31407fe76f65c2173b"
server_key_version: "encryptionkey@0"
I20260925 04:37:40.698225 16514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:40.710105 16514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:40.713266 16514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:40.713488 16514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:40.721251 16514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40935
I20260925 04:37:40.721280 16583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40935 every 8 connection(s)
I20260925 04:37:40.722323 16514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:40.723520 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16514
I20260925 04:37:40.723671 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:40.723975 27835 external_mini_cluster.cc:1428] Setting key ef7a4a2a0a6d5555b9d64b0aacc6c969
I20260925 04:37:40.725602 16584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:40.732892 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: Bootstrap starting.
I20260925 04:37:40.734474 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:40.725473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41415 (local address 127.0.0.1:40935)
0925 04:37:40.725992 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:40.726012 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:40.726056 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:37:40.726889 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:40.726913 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:40.727368 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:40.727734 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:40.728136 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:40.729717 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:40.730609 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:40.731358 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:40.731536 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:40.732365 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:40.732371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:40.732388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:40.732886 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:40.732959 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:40.733001 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:40.733097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:40.733268 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 04:37:40.733479 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":164,"threads_started":1}
I20260925 04:37:40.735575 16584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:40.736356 16584 log.cc:824] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:40.737557 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:40.738857 16584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: No bootstrap required, opened a new log
I20260925 04:37:40.743499 16584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260925 04:37:40.743845 16584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:40.743916 16584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa7395a910740e180e7c81d29cb7f8f, State: Initialized, Role: FOLLOWER
I20260925 04:37:40.744216 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:40.739850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52253 (local address 127.0.0.1:40935)
0925 04:37:40.739989 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:40.739993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:40.740163 (+ 170us) server_negotiation.cc:407] Connection header received
0925 04:37:40.740286 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:40.740289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:40.740329 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:40.740393 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:40.740690 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:40.741195 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:40.742479 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:40.742728 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:40.742808 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:40.743683 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:40.743686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:40.743688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:40.743700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:40.743724 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:40.743734 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:40.743801 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:40.743971 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 04:37:40.744051 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:37:40.744490 16584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260925 04:37:40.744666 16584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:40.744750 16584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:40.744841 16584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:40.745946 16584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260925 04:37:40.746254 16584 leader_election.cc:302] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fa7395a910740e180e7c81d29cb7f8f; no voters:
I20260925 04:37:40.746512 16584 leader_election.cc:288] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:40.746616 16594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:40.746905 16594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [term 1 LEADER]: Becoming Leader. State: Replica: 6fa7395a910740e180e7c81d29cb7f8f, State: Running, Role: LEADER
I20260925 04:37:40.747254 16594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } }
I20260925 04:37:40.747861 16584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:40.749573 16596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fa7395a910740e180e7c81d29cb7f8f. Latest consensus state: current_term: 1 leader_uuid: "6fa7395a910740e180e7c81d29cb7f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260925 04:37:40.749749 16596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:40.749991 16595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fa7395a910740e180e7c81d29cb7f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fa7395a910740e180e7c81d29cb7f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40935 } } }
I20260925 04:37:40.750190 16595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f [sys.catalog]: This master's current role is: LEADER
W20260925 04:37:40.752771 16609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:37:40.752839 16609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:37:40.752921 16608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:40.753732 16608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:40.763759 16608 catalog_manager.cc:1347] Generated new cluster ID: aab03f4357b54e168b42b329e7036342
I20260925 04:37:40.763860 16608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:40.776237 16608 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:40.777810 16608 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:40.792384 16608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fa7395a910740e180e7c81d29cb7f8f: Generated new TSK 0
I20260925 04:37:40.793164 16608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:40.806221 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16514
2026-09-25T04:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:40 dist-test-slave-r8r3 krb5kdc[16615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:40 dist-test-slave-r8r3 krb5kdc[16615](info): set up 2 sockets
Sep 25 04:37:40 dist-test-slave-r8r3 krb5kdc[16615](info): commencing operation
krb5kdc: starting...
W20260925 04:37:41.854334 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:41 dist-test-slave-r8r3 krb5kdc[16615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:41.978322 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:42.090880 16628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:42.091179 16628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:42.091257 16628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:42.095284 16628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:42.095386 16628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:42.095424 16628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:42.095495 16628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:42.095534 16628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:42.100888 16628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:42.102065 16628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:42.103128 16628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:42.111670 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:42.111656 16628 server_base.cc:1034] running on GCE node
W20260925 04:37:42.111644 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:42.111688 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:42.112435 16628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:42.113602 16628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:42.114782 16628 hybrid_clock.cc:648] HybridClock initialized: now 1790311062114765 us; error 30 us; skew 500 ppm
Sep 25 04:37:42 dist-test-slave-r8r3 krb5kdc[16615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:42.117453 16628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:42.118443 16628 webserver.cc:460] Webserver started at http://127.0.0.1:41255/ using document root <none> and password file <none>
I20260925 04:37:42.119011 16628 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:42.119102 16628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:42.119345 16628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:42.121373 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "1499f0780a534457b99edba99b4866b0"
format_stamp: "Formatted at 2026-09-25 04:37:42 on dist-test-slave-r8r3"
server_key: "609e24e1e87c51737bb464c7c1d7c3de"
server_key_iv: "e4c62c1f02842c10b1ea149064b0a55f"
server_key_version: "encryptionkey@0"
I20260925 04:37:42.121981 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "1499f0780a534457b99edba99b4866b0"
format_stamp: "Formatted at 2026-09-25 04:37:42 on dist-test-slave-r8r3"
server_key: "609e24e1e87c51737bb464c7c1d7c3de"
server_key_iv: "e4c62c1f02842c10b1ea149064b0a55f"
server_key_version: "encryptionkey@0"
I20260925 04:37:42.125756 16628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:37:42.128167 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:42.129364 16628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:42.129554 16628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "1499f0780a534457b99edba99b4866b0"
format_stamp: "Formatted at 2026-09-25 04:37:42 on dist-test-slave-r8r3"
server_key: "609e24e1e87c51737bb464c7c1d7c3de"
server_key_iv: "e4c62c1f02842c10b1ea149064b0a55f"
server_key_version: "encryptionkey@0"
I20260925 04:37:42.129674 16628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:42.134481 16628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:42.137744 16628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:42.137938 16628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:42.145259 16628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39617
I20260925 04:37:42.145288 16696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39617 every 8 connection(s)
I20260925 04:37:42.146240 16628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:42.148972 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:42.154633 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0: Bootstrap starting.
I20260925 04:37:42.155107 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16628
I20260925 04:37:42.155243 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:42.155551 27835 external_mini_cluster.cc:1428] Setting key 4ab40ecbc2567b59519e4eedebfde9f4
I20260925 04:37:42.157682 16697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:42.158427 16697 log.cc:824] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:42.160496 16697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0: No bootstrap required, opened a new log
I20260925 04:37:42.161057 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:42.157085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53155 (local address 127.0.0.1:39617)
0925 04:37:42.157628 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:42.157643 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:42.157677 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:37:42.158364 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:42.158384 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:42.158718 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:42.159047 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:42.159834 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:42.159838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:42.159849 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:42.159858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:42.159906 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:42.159924 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:42.160031 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:42.160120 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:37:42.160275 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":163,"threads_started":1}
I20260925 04:37:42.164179 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:42.164867 16697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } }
I20260925 04:37:42.165087 16697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:42.165151 16697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1499f0780a534457b99edba99b4866b0, State: Initialized, Role: FOLLOWER
I20260925 04:37:42.165643 16697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } }
I20260925 04:37:42.165807 16697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:42.165874 16697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:42.165971 16697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:42.167282 16697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } }
I20260925 04:37:42.167681 16697 leader_election.cc:302] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1499f0780a534457b99edba99b4866b0; no voters:
I20260925 04:37:42.168056 16697 leader_election.cc:288] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:42.168174 16708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:42.168222 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:42.166423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57213 (local address 127.0.0.1:39617)
0925 04:37:42.166543 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:42.166547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:42.166710 (+ 163us) server_negotiation.cc:407] Connection header received
0925 04:37:42.166825 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:42.166827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:42.166856 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:42.166919 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:42.167668 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:42.167671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:42.167673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:42.167683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:42.167707 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:42.167717 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:42.167785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:42.168064 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 04:37:42.168107 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:37:42.168622 16708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1499f0780a534457b99edba99b4866b0, State: Running, Role: LEADER
I20260925 04:37:42.168967 16708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } }
I20260925 04:37:42.169440 16697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:42.170246 16708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1499f0780a534457b99edba99b4866b0. Latest consensus state: current_term: 1 leader_uuid: "1499f0780a534457b99edba99b4866b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } } }
I20260925 04:37:42.170378 16708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:42.170787 16718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:42.171197 16709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1499f0780a534457b99edba99b4866b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1499f0780a534457b99edba99b4866b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39617 } } }
I20260925 04:37:42.171368 16709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:42.174012 16718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:42.179641 16718 catalog_manager.cc:1347] Generated new cluster ID: 67112f2b54284b619b48c845a8e67188
I20260925 04:37:42.179723 16718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:42.198078 16718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:42.198480 16718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:42.207192 16718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1499f0780a534457b99edba99b4866b0: Generated new TSK 0
I20260925 04:37:42.207976 16718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:42.223174 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16628
2026-09-25T04:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:42 dist-test-slave-r8r3 krb5kdc[16730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:42 dist-test-slave-r8r3 krb5kdc[16730](info): set up 2 sockets
Sep 25 04:37:42 dist-test-slave-r8r3 krb5kdc[16730](info): commencing operation
krb5kdc: starting...
W20260925 04:37:43.281877 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:43.412434 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:43.526603 16743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:43.526959 16743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:43.527055 16743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:43.531289 16743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:43.531388 16743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:43.531427 16743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:43.531487 16743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:43.531528 16743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:43.536953 16743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:43.538192 16743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:43.539299 16743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:43.548338 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:43.548390 16743 server_base.cc:1034] running on GCE node
W20260925 04:37:43.548624 16748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:43.548396 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:43.549335 16743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:43.550566 16743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:43.551772 16743 hybrid_clock.cc:648] HybridClock initialized: now 1790311063551761 us; error 38 us; skew 500 ppm
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:43.554639 16743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:43.555586 16743 webserver.cc:460] Webserver started at http://127.0.0.1:41935/ using document root <none> and password file <none>
I20260925 04:37:43.556176 16743 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:43.556280 16743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:43.556519 16743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:43.558734 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "da57c53119424cf69850108af71684e7"
format_stamp: "Formatted at 2026-09-25 04:37:43 on dist-test-slave-r8r3"
server_key: "558201aedd2ff7a930cab375564f563b"
server_key_iv: "49d42187430b5b19b17945f78dc15f44"
server_key_version: "encryptionkey@0"
I20260925 04:37:43.559331 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "da57c53119424cf69850108af71684e7"
format_stamp: "Formatted at 2026-09-25 04:37:43 on dist-test-slave-r8r3"
server_key: "558201aedd2ff7a930cab375564f563b"
server_key_iv: "49d42187430b5b19b17945f78dc15f44"
server_key_version: "encryptionkey@0"
I20260925 04:37:43.563344 16743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:43.565807 16758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:43.567036 16743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:37:43.567195 16743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "da57c53119424cf69850108af71684e7"
format_stamp: "Formatted at 2026-09-25 04:37:43 on dist-test-slave-r8r3"
server_key: "558201aedd2ff7a930cab375564f563b"
server_key_iv: "49d42187430b5b19b17945f78dc15f44"
server_key_version: "encryptionkey@0"
I20260925 04:37:43.567327 16743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:43.594117 16743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:43.597237 16743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:43.597474 16743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:43.604481 16743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40745
I20260925 04:37:43.604578 16812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40745 every 8 connection(s)
I20260925 04:37:43.605573 16743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:43.608439 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:43.609716 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16743
I20260925 04:37:43.609872 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:43.610211 27835 external_mini_cluster.cc:1428] Setting key 7fa82b84f705dd831ae0995f7c657c11
I20260925 04:37:43.614970 16813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7: Bootstrap starting.
I20260925 04:37:43.617290 16813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:43.618144 16813 log.cc:824] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:37:43.620507 16813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7: No bootstrap required, opened a new log
I20260925 04:37:43.625231 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:43.611641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:51427 (local address 127.0.0.1:40745)
0925 04:37:43.612131 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:43.612147 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:43.612181 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:37:43.612818 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:43.612835 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:43.613192 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:43.613577 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:43.614447 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:43.615493 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:43.616383 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:43.616869 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:43.617053 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:43.619794 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:43.619801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:43.619818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:43.619849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:43.621806 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:43.622308 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:43.622314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:43.622322 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:43.622461 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:43.622796 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:43.622799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:43.622801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:43.623443 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:43.623650 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:43.623918 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:37:43.624155 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":145,"threads_started":1}
I20260925 04:37:43.625557 16813 raft_consensus.cc:348] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 04:37:43.625851 16813 raft_consensus.cc:374] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:43.625911 16813 raft_consensus.cc:729] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da57c53119424cf69850108af71684e7, State: Initialized, Role: FOLLOWER
I20260925 04:37:43.626436 16813 consensus_queue.cc:260] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 04:37:43.626598 16813 raft_consensus.cc:388] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:43.626663 16813 raft_consensus.cc:482] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:43.626791 16813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:43.628073 16813 raft_consensus.cc:504] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 04:37:43.628474 16813 leader_election.cc:302] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da57c53119424cf69850108af71684e7; no voters:
I20260925 04:37:43.628530 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:43.628834 16813 leader_election.cc:288] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:43.628969 16818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:43.629209 16818 raft_consensus.cc:686] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [term 1 LEADER]: Becoming Leader. State: Replica: da57c53119424cf69850108af71684e7, State: Running, Role: LEADER
I20260925 04:37:43.629626 16818 consensus_queue.cc:237] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } }
I20260925 04:37:43.630239 16813 sys_catalog.cc:564] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:43.631717 16820 sys_catalog.cc:455] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da57c53119424cf69850108af71684e7. Latest consensus state: current_term: 1 leader_uuid: "da57c53119424cf69850108af71684e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260925 04:37:43.631876 16820 sys_catalog.cc:458] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:43.631783 16819 sys_catalog.cc:455] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da57c53119424cf69850108af71684e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da57c53119424cf69850108af71684e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40745 } } }
I20260925 04:37:43.631973 16819 sys_catalog.cc:458] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:43.632355 16830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:43.635867 16830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:43.641263 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:43.630885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:45459 (local address 127.0.0.1:40745)
0925 04:37:43.630996 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:43.631000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:43.631522 (+ 522us) server_negotiation.cc:407] Connection header received
0925 04:37:43.631698 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:43.631701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:43.631754 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:43.631839 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:43.632822 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:43.633655 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:43.634781 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:43.635135 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:43.635239 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:43.637000 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:43.637006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:43.637011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:43.637039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:43.638821 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:43.639505 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:43.639509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:43.639511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:43.639599 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:43.640022 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:43.640025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:43.640027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:43.640247 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:43.640378 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:43.640834 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 04:37:43.641139 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:37:43.642925 16830 catalog_manager.cc:1347] Generated new cluster ID: 6e534eb0efcc4ba5a969b0d7f559e903
I20260925 04:37:43.642997 16830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:43.650259 16830 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:43.651446 16830 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:43.656296 16830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da57c53119424cf69850108af71684e7: Generated new TSK 0
I20260925 04:37:43.657090 16830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:43.668916 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16743
W20260925 04:37:43.675249 16728 connection.cc:535] client connection to 127.0.0.1:40745 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40745): Connection reset by peer (error 104)
2026-09-25T04:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16844](info): set up 2 sockets
Sep 25 04:37:43 dist-test-slave-r8r3 krb5kdc[16844](info): commencing operation
krb5kdc: starting...
W20260925 04:37:44.726002 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:44 dist-test-slave-r8r3 krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:44.850147 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:44.965837 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:44.966156 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:44.966200 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:44.970562 16858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:44.970640 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:44.970660 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:44.970677 16858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:44.970693 16858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:44.976737 16858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:44.978093 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:44.979231 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:44.988478 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:44.988502 16863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:44.988488 16864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:44.988546 16858 server_base.cc:1034] running on GCE node
I20260925 04:37:44.989398 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:44.990621 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:44.991822 16858 hybrid_clock.cc:648] HybridClock initialized: now 1790311064991805 us; error 35 us; skew 500 ppm
Sep 25 04:37:44 dist-test-slave-r8r3 krb5kdc[16844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:44.998729 16858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:45.000044 16858 webserver.cc:460] Webserver started at http://127.0.0.1:37257/ using document root <none> and password file <none>
I20260925 04:37:45.000631 16858 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:45.000697 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:45.000877 16858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:45.003116 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "f59e224207e849cc8bc56fc0cbe2ac4c"
format_stamp: "Formatted at 2026-09-25 04:37:44 on dist-test-slave-r8r3"
server_key: "253c1aecf178dcd1ecc0abb8595bc85d"
server_key_iv: "f850218a478242c5e7d5d1f7d950a0da"
server_key_version: "encryptionkey@0"
I20260925 04:37:45.003712 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "f59e224207e849cc8bc56fc0cbe2ac4c"
format_stamp: "Formatted at 2026-09-25 04:37:44 on dist-test-slave-r8r3"
server_key: "253c1aecf178dcd1ecc0abb8595bc85d"
server_key_iv: "f850218a478242c5e7d5d1f7d950a0da"
server_key_version: "encryptionkey@0"
I20260925 04:37:45.007733 16858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:45.010365 16873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:45.011783 16858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:45.011948 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "f59e224207e849cc8bc56fc0cbe2ac4c"
format_stamp: "Formatted at 2026-09-25 04:37:44 on dist-test-slave-r8r3"
server_key: "253c1aecf178dcd1ecc0abb8595bc85d"
server_key_iv: "f850218a478242c5e7d5d1f7d950a0da"
server_key_version: "encryptionkey@0"
I20260925 04:37:45.012075 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:45.047283 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:45.050854 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:45.051074 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:45.058684 16858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37653
I20260925 04:37:45.058729 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37653 every 8 connection(s)
I20260925 04:37:45.059719 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:45.062513 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:45.067627 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16858
I20260925 04:37:45.067759 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:45.068054 27835 external_mini_cluster.cc:1428] Setting key 0f1630c6db52f6fbc6ea81927371e277
I20260925 04:37:45.068450 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c: Bootstrap starting.
I20260925 04:37:45.071264 16926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:45.072176 16926 log.cc:824] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:45.074626 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c: No bootstrap required, opened a new log
I20260925 04:37:45.079092 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:45.069646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:51919 (local address 127.0.0.1:37653)
0925 04:37:45.070098 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:45.070118 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:45.070162 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:37:45.071014 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:45.071038 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:45.071534 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:45.071908 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:45.072364 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:45.073757 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:45.075160 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:45.075978 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:45.076201 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:45.077159 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:45.077166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:45.077185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:45.077774 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:45.077834 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:45.077859 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:45.078008 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:45.078175 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 04:37:45.078324 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":129,"threads_started":1}
I20260925 04:37:45.079473 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260925 04:37:45.079730 16926 raft_consensus.cc:374] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:45.079813 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59e224207e849cc8bc56fc0cbe2ac4c, State: Initialized, Role: FOLLOWER
I20260925 04:37:45.080225 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260925 04:37:45.080374 16926 raft_consensus.cc:388] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:45.080456 16926 raft_consensus.cc:482] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:45.080554 16926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:45.081717 16926 raft_consensus.cc:504] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260925 04:37:45.082089 16926 leader_election.cc:302] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f59e224207e849cc8bc56fc0cbe2ac4c; no voters:
I20260925 04:37:45.082300 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:45.082479 16926 leader_election.cc:288] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:45.082562 16932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:45.082803 16932 raft_consensus.cc:686] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [term 1 LEADER]: Becoming Leader. State: Replica: f59e224207e849cc8bc56fc0cbe2ac4c, State: Running, Role: LEADER
I20260925 04:37:45.083213 16932 consensus_queue.cc:237] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260925 04:37:45.083863 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:45.084546 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59e224207e849cc8bc56fc0cbe2ac4c. Latest consensus state: current_term: 1 leader_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260925 04:37:45.084674 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:45.084993 16933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59e224207e849cc8bc56fc0cbe2ac4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260925 04:37:45.085073 16944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:45.085091 16933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:45.088528 16944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:45.091043 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:45.084750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:57139 (local address 127.0.0.1:37653)
0925 04:37:45.084867 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:45.084871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:45.086177 (+ 1306us) server_negotiation.cc:407] Connection header received
0925 04:37:45.086368 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:45.086371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:45.086435 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:45.086523 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:45.086816 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:45.087558 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:45.089054 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:45.089414 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:45.089545 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:45.090574 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:45.090577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:45.090580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:45.090594 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:45.090624 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:45.090651 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:45.090738 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:45.090859 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 04:37:45.090891 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:37:45.095597 16944 catalog_manager.cc:1347] Generated new cluster ID: ae15ac98065f4a0190a29baec6278755
I20260925 04:37:45.095685 16944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:45.104777 16944 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:45.106134 16944 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:45.115509 16944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f59e224207e849cc8bc56fc0cbe2ac4c: Generated new TSK 0
I20260925 04:37:45.116425 16944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:45.127920 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16858
W20260925 04:37:45.134363 16842 connection.cc:535] client connection to 127.0.0.1:37653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37653): Connection reset by peer (error 104)
2026-09-25T04:37:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:45 dist-test-slave-r8r3 krb5kdc[16958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:45 dist-test-slave-r8r3 krb5kdc[16958](info): set up 2 sockets
Sep 25 04:37:45 dist-test-slave-r8r3 krb5kdc[16958](info): commencing operation
krb5kdc: starting...
W20260925 04:37:46.173894 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:46 dist-test-slave-r8r3 krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:46.312047 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:46.431282 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:46.431597 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:46.431643 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:46.435745 16971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:46.435822 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:46.435842 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:46.435859 16971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:46.435874 16971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:46.442283 16971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:46.443475 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:46.444502 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:46.453476 16979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:46.453514 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:46.453478 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:46.453873 16971 server_base.cc:1034] running on GCE node
I20260925 04:37:46.454584 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:46.455824 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:46.457022 16971 hybrid_clock.cc:648] HybridClock initialized: now 1790311066457011 us; error 36 us; skew 500 ppm
Sep 25 04:37:46 dist-test-slave-r8r3 krb5kdc[16958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:46.459997 16971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:46.460960 16971 webserver.cc:460] Webserver started at http://127.0.0.1:39379/ using document root <none> and password file <none>
I20260925 04:37:46.461777 16971 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:46.461886 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:46.462188 16971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:46.464475 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "c35a5b16f8d2402bbe53c0643d75daef"
format_stamp: "Formatted at 2026-09-25 04:37:46 on dist-test-slave-r8r3"
server_key: "ba8f94c0a02342e049d5f0a3acfa2d76"
server_key_iv: "76a329cf9388a3141ac8fff1224312a7"
server_key_version: "encryptionkey@0"
I20260925 04:37:46.465034 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "c35a5b16f8d2402bbe53c0643d75daef"
format_stamp: "Formatted at 2026-09-25 04:37:46 on dist-test-slave-r8r3"
server_key: "ba8f94c0a02342e049d5f0a3acfa2d76"
server_key_iv: "76a329cf9388a3141ac8fff1224312a7"
server_key_version: "encryptionkey@0"
I20260925 04:37:46.468891 16971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:37:46.471692 16986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:46.473097 16971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:37:46.473215 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "c35a5b16f8d2402bbe53c0643d75daef"
format_stamp: "Formatted at 2026-09-25 04:37:46 on dist-test-slave-r8r3"
server_key: "ba8f94c0a02342e049d5f0a3acfa2d76"
server_key_iv: "76a329cf9388a3141ac8fff1224312a7"
server_key_version: "encryptionkey@0"
I20260925 04:37:46.473316 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:46.478098 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:46.481097 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:46.481276 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:46.488740 16971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36397
I20260925 04:37:46.488757 17040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36397 every 8 connection(s)
I20260925 04:37:46.489794 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:46.492786 17041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:46.498899 17041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef: Bootstrap starting.
I20260925 04:37:46.499528 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 16971
I20260925 04:37:46.499686 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:46.499981 27835 external_mini_cluster.cc:1428] Setting key 90a5beea8a0968ca63ffda8986d0075c
I20260925 04:37:46.502079 17041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:46.502997 17041 log.cc:824] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:46.505484 17041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef: No bootstrap required, opened a new log
I20260925 04:37:46.506093 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:46.501598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:38327 (local address 127.0.0.1:36397)
0925 04:37:46.502078 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:46.502099 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:46.502145 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:37:46.503062 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:46.503090 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:46.503515 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:46.503825 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:46.504656 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:46.504670 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:46.504684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:46.504697 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:46.504761 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:46.504787 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:46.504900 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:46.504972 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 04:37:46.505130 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260925 04:37:46.509054 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:46.509963 17041 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } }
I20260925 04:37:46.510211 17041 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:46.510283 17041 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35a5b16f8d2402bbe53c0643d75daef, State: Initialized, Role: FOLLOWER
I20260925 04:37:46.510790 17041 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } }
I20260925 04:37:46.510929 17041 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:46.511022 17041 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:46.511113 17041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:46.512382 17041 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } }
I20260925 04:37:46.512837 17041 leader_election.cc:302] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35a5b16f8d2402bbe53c0643d75daef; no voters:
I20260925 04:37:46.513080 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:46.511347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:36711 (local address 127.0.0.1:36397)
0925 04:37:46.511497 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:46.511502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:46.511680 (+ 178us) server_negotiation.cc:407] Connection header received
0925 04:37:46.511765 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:46.511768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:46.511807 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:46.511894 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:46.512681 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:46.512684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:46.512687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:46.512700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:46.512730 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:46.512743 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:46.512850 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:46.512912 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 04:37:46.512956 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:37:46.513276 17041 leader_election.cc:288] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:46.513360 17052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:46.513705 17052 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [term 1 LEADER]: Becoming Leader. State: Replica: c35a5b16f8d2402bbe53c0643d75daef, State: Running, Role: LEADER
I20260925 04:37:46.514109 17052 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } }
I20260925 04:37:46.514724 17041 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:46.515342 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35a5b16f8d2402bbe53c0643d75daef. Latest consensus state: current_term: 1 leader_uuid: "c35a5b16f8d2402bbe53c0643d75daef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } } }
I20260925 04:37:46.515523 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:46.515870 17053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35a5b16f8d2402bbe53c0643d75daef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35a5b16f8d2402bbe53c0643d75daef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36397 } } }
I20260925 04:37:46.515976 17060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:46.516054 17053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:46.519472 17060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:46.526043 17060 catalog_manager.cc:1347] Generated new cluster ID: fb69d5eb43da42c982fdb9de80f861de
I20260925 04:37:46.526146 17060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:46.546022 17060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:46.546416 17060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:46.557715 17060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35a5b16f8d2402bbe53c0643d75daef: Generated new TSK 0
I20260925 04:37:46.558563 17060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:46.565770 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 16971
2026-09-25T04:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:46 dist-test-slave-r8r3 krb5kdc[17073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:46 dist-test-slave-r8r3 krb5kdc[17073](info): set up 2 sockets
Sep 25 04:37:46 dist-test-slave-r8r3 krb5kdc[17073](info): commencing operation
krb5kdc: starting...
W20260925 04:37:47.630064 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:47 dist-test-slave-r8r3 krb5kdc[17073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:47.754467 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:47.865306 17086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:47.865648 17086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:47.865727 17086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:47.869750 17086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:47.869854 17086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:47.869894 17086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:47.869930 17086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:47.869966 17086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:47.875217 17086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:47.876319 17086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:47.877389 17086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:37:47.886004 17086 server_base.cc:1034] running on GCE node
W20260925 04:37:47.885970 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:47.885970 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:47.885989 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:47.886828 17086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:47.887914 17086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:47.889101 17086 hybrid_clock.cc:648] HybridClock initialized: now 1790311067889086 us; error 33 us; skew 500 ppm
Sep 25 04:37:47 dist-test-slave-r8r3 krb5kdc[17073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:47.891855 17086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:47.892689 17086 webserver.cc:460] Webserver started at http://127.0.0.1:44065/ using document root <none> and password file <none>
I20260925 04:37:47.893249 17086 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:47.893338 17086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:47.893627 17086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:47.895628 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "75e4d343bd8b46848327cb777ca2f36b"
format_stamp: "Formatted at 2026-09-25 04:37:47 on dist-test-slave-r8r3"
server_key: "8a8731a83319ab2a7fa16e0aa9d13d86"
server_key_iv: "802d9a99d8e76b56196b16853d0ca54a"
server_key_version: "encryptionkey@0"
I20260925 04:37:47.896209 17086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "75e4d343bd8b46848327cb777ca2f36b"
format_stamp: "Formatted at 2026-09-25 04:37:47 on dist-test-slave-r8r3"
server_key: "8a8731a83319ab2a7fa16e0aa9d13d86"
server_key_iv: "802d9a99d8e76b56196b16853d0ca54a"
server_key_version: "encryptionkey@0"
I20260925 04:37:47.900015 17086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 04:37:47.902545 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:47.903788 17086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:47.903960 17086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "75e4d343bd8b46848327cb777ca2f36b"
format_stamp: "Formatted at 2026-09-25 04:37:47 on dist-test-slave-r8r3"
server_key: "8a8731a83319ab2a7fa16e0aa9d13d86"
server_key_iv: "802d9a99d8e76b56196b16853d0ca54a"
server_key_version: "encryptionkey@0"
I20260925 04:37:47.904086 17086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:47.916411 17086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:47.919306 17086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:47.919533 17086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:47.927166 17086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46239
I20260925 04:37:47.927201 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46239 every 8 connection(s)
I20260925 04:37:47.928148 17086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:47.930924 17155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:47.931097 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17086
I20260925 04:37:47.931210 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:47.931456 27835 external_mini_cluster.cc:1428] Setting key a0ad1b8219338100558b442083fb17ac
I20260925 04:37:47.937902 17155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b: Bootstrap starting.
I20260925 04:37:47.940307 17155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b: Neither blocks nor log segments found. Creating new log.
Sep 25 04:37:47 dist-test-slave-r8r3 krb5kdc[17073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:37:47.941259 17155 log.cc:824] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:47.943257 17155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b: No bootstrap required, opened a new log
I20260925 04:37:47.946657 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:47.932857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:43395 (local address 127.0.0.1:46239)
0925 04:37:47.933308 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:47.933328 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:47.933369 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:37:47.934215 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:47.934239 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:47.934666 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:47.935015 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:47.935670 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:47.937157 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:47.938082 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:47.938551 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:47.938749 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:47.941669 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:47.941677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:47.941713 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:47.941751 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:47.943537 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:47.944103 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:47.944107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:47.944112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:47.944221 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:47.944604 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:47.944606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:47.944607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:47.945001 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:47.945158 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:47.945597 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 04:37:47.945851 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260925 04:37:47.947539 17155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260925 04:37:47.947829 17155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:47.947893 17155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e4d343bd8b46848327cb777ca2f36b, State: Initialized, Role: FOLLOWER
I20260925 04:37:47.948331 17155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260925 04:37:47.948489 17155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:47.948559 17155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:47.948683 17155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:47.949877 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:47.949908 17155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260925 04:37:47.950244 17155 leader_election.cc:302] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75e4d343bd8b46848327cb777ca2f36b; no voters:
I20260925 04:37:47.950630 17155 leader_election.cc:288] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:47.950759 17160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:47.950980 17160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [term 1 LEADER]: Becoming Leader. State: Replica: 75e4d343bd8b46848327cb777ca2f36b, State: Running, Role: LEADER
I20260925 04:37:47.951363 17160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } }
I20260925 04:37:47.952029 17155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:47.952613 17160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75e4d343bd8b46848327cb777ca2f36b. Latest consensus state: current_term: 1 leader_uuid: "75e4d343bd8b46848327cb777ca2f36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260925 04:37:47.952891 17160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:47.953052 17161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75e4d343bd8b46848327cb777ca2f36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e4d343bd8b46848327cb777ca2f36b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46239 } } }
I20260925 04:37:47.953164 17161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:47.953372 17173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:47.956496 17173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:47.962809 17173 catalog_manager.cc:1347] Generated new cluster ID: d9dc8d760052411fa3922989e136d03b
I20260925 04:37:47.962869 17173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:47.963570 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:47.952116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:37851 (local address 127.0.0.1:46239)
0925 04:37:47.952242 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:47.952246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:47.953857 (+ 1611us) server_negotiation.cc:407] Connection header received
0925 04:37:47.954135 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:47.954138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:47.954198 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:47.954268 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:47.955275 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:47.955956 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:47.957483 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:47.957843 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:47.957948 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:47.959651 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:47.959655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:47.959658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:47.959684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:47.961439 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:47.962020 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:47.962023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:47.962025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:47.962118 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:47.962528 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:47.962532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:47.962533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:47.962750 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:47.962871 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:47.963293 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 04:37:47.963427 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:37:47.990738 17173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:47.991923 17173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:48.002149 17173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75e4d343bd8b46848327cb777ca2f36b: Generated new TSK 0
I20260925 04:37:48.002985 17173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:48.008662 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17086
W20260925 04:37:48.014482 17071 connection.cc:535] client connection to 127.0.0.1:46239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46239): Connection reset by peer (error 104)
2026-09-25T04:37:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:48 dist-test-slave-r8r3 krb5kdc[17187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:48 dist-test-slave-r8r3 krb5kdc[17187](info): set up 2 sockets
Sep 25 04:37:48 dist-test-slave-r8r3 krb5kdc[17187](info): commencing operation
krb5kdc: starting...
W20260925 04:37:49.049849 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:49 dist-test-slave-r8r3 krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:49.175768 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:49.290810 17200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:49.291096 17200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:49.291137 17200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:49.295233 17200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:49.295310 17200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:49.295329 17200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:49.295346 17200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:49.295361 17200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:49.300786 17200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44947
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:49.301963 17200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:49.303118 17200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:49.312477 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:49.312494 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:49.312529 17200 server_base.cc:1034] running on GCE node
W20260925 04:37:49.312494 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:49.313346 17200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:49.314700 17200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:49.315896 17200 hybrid_clock.cc:648] HybridClock initialized: now 1790311069315868 us; error 40 us; skew 500 ppm
Sep 25 04:37:49 dist-test-slave-r8r3 krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:49.318761 17200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:49.319664 17200 webserver.cc:460] Webserver started at http://127.0.0.1:34379/ using document root <none> and password file <none>
I20260925 04:37:49.320271 17200 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:49.320334 17200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:49.320516 17200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:49.322713 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "0414d6d29419421e91712d3c06c43844"
format_stamp: "Formatted at 2026-09-25 04:37:49 on dist-test-slave-r8r3"
server_key: "18307fb3c0ea21cf3ff0afd2b4ae3cde"
server_key_iv: "1c4cf844a79510b49d6950e3c775d27d"
server_key_version: "encryptionkey@0"
I20260925 04:37:49.323235 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "0414d6d29419421e91712d3c06c43844"
format_stamp: "Formatted at 2026-09-25 04:37:49 on dist-test-slave-r8r3"
server_key: "18307fb3c0ea21cf3ff0afd2b4ae3cde"
server_key_iv: "1c4cf844a79510b49d6950e3c775d27d"
server_key_version: "encryptionkey@0"
I20260925 04:37:49.327117 17200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:37:49.329671 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:49.331045 17200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:37:49.331233 17200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "0414d6d29419421e91712d3c06c43844"
format_stamp: "Formatted at 2026-09-25 04:37:49 on dist-test-slave-r8r3"
server_key: "18307fb3c0ea21cf3ff0afd2b4ae3cde"
server_key_iv: "1c4cf844a79510b49d6950e3c775d27d"
server_key_version: "encryptionkey@0"
I20260925 04:37:49.331342 17200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:49.344347 17200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:49.347793 17200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:49.348019 17200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:49.355439 17200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44947
I20260925 04:37:49.355465 17270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44947 every 8 connection(s)
I20260925 04:37:49.356415 17200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:49.359205 17271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:49.362332 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17200
I20260925 04:37:49.362473 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:49.362753 27835 external_mini_cluster.cc:1428] Setting key 321a5599eac00be515da85f89e8416f4
I20260925 04:37:49.365339 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844: Bootstrap starting.
I20260925 04:37:49.368146 17271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:49.368994 17271 log.cc:824] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:49.371448 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844: No bootstrap required, opened a new log
I20260925 04:37:49.373057 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:49.364244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:58277 (local address 127.0.0.1:44947)
0925 04:37:49.364692 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:49.364711 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:49.364752 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:37:49.365597 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:49.365625 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:49.366045 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:49.366411 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:49.366852 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:49.368325 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:49.369300 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:49.369985 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:49.370157 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:49.371015 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:49.371021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:49.371037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:49.371568 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:49.371637 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:49.371668 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:49.371798 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:49.371979 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 04:37:49.372123 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
I20260925 04:37:49.375979 17271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 04:37:49.376183 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:49.376312 17271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:49.376375 17271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0414d6d29419421e91712d3c06c43844, State: Initialized, Role: FOLLOWER
I20260925 04:37:49.376869 17271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 04:37:49.377028 17271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:49.377106 17271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:49.377206 17271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:49.378500 17271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 04:37:49.378873 17271 leader_election.cc:302] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0414d6d29419421e91712d3c06c43844; no voters:
I20260925 04:37:49.379256 17271 leader_election.cc:288] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:49.379362 17281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:49.379597 17281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [term 1 LEADER]: Becoming Leader. State: Replica: 0414d6d29419421e91712d3c06c43844, State: Running, Role: LEADER
I20260925 04:37:49.380009 17281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } }
I20260925 04:37:49.380728 17271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:49.381771 17282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0414d6d29419421e91712d3c06c43844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260925 04:37:49.381904 17282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:49.382256 17283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0414d6d29419421e91712d3c06c43844. Latest consensus state: current_term: 1 leader_uuid: "0414d6d29419421e91712d3c06c43844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0414d6d29419421e91712d3c06c43844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44947 } } }
I20260925 04:37:49.382391 17283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:49.383279 17290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:49.383867 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:49.378563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:45265 (local address 127.0.0.1:44947)
0925 04:37:49.378694 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:49.378698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:49.378934 (+ 236us) server_negotiation.cc:407] Connection header received
0925 04:37:49.379019 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:49.379022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:49.379130 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:49.379225 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:49.379598 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:49.380387 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:49.381568 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:49.381912 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:49.381992 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:49.383273 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:49.383278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:49.383281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:49.383296 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:49.383340 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:49.383382 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:49.383449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:49.383662 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 04:37:49.383695 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":51}
I20260925 04:37:49.386595 17290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:49.392522 17290 catalog_manager.cc:1347] Generated new cluster ID: 0cdfd82b8e634d0cbf52ea419a8b0a22
I20260925 04:37:49.392613 17290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:49.400190 17290 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:49.401114 17290 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:49.413405 17290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0414d6d29419421e91712d3c06c43844: Generated new TSK 0
I20260925 04:37:49.414471 17290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:49.426364 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17200
W20260925 04:37:49.432801 17185 connection.cc:535] client connection to 127.0.0.1:44947 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44947): Connection reset by peer (error 104)
2026-09-25T04:37:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:49 dist-test-slave-r8r3 krb5kdc[17302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:49 dist-test-slave-r8r3 krb5kdc[17302](info): set up 2 sockets
Sep 25 04:37:49 dist-test-slave-r8r3 krb5kdc[17302](info): commencing operation
krb5kdc: starting...
W20260925 04:37:50.457274 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:50 dist-test-slave-r8r3 krb5kdc[17302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:37:50.579931 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:50.692606 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:50.692916 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:50.692958 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:50.696977 17316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:50.697037 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:50.697057 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:50.697073 17316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:50.697088 17316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:50.702412 17316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:50.703495 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:50.704499 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:50.713251 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:50.713299 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:50.713404 17316 server_base.cc:1034] running on GCE node
W20260925 04:37:50.713567 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:50.714171 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:50.715377 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:50.716581 17316 hybrid_clock.cc:648] HybridClock initialized: now 1790311070716566 us; error 24 us; skew 500 ppm
Sep 25 04:37:50 dist-test-slave-r8r3 krb5kdc[17302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:50.722761 17316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:37:50.723717 17316 webserver.cc:460] Webserver started at http://127.0.0.1:32927/ using document root <none> and password file <none>
I20260925 04:37:50.724285 17316 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:50.724359 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:50.724538 17316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:50.726729 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "968de3ef59c3489c8a717de1b3de8bec"
format_stamp: "Formatted at 2026-09-25 04:37:50 on dist-test-slave-r8r3"
server_key: "1fdbe4824962e94c8fcde0ebcba779c2"
server_key_iv: "5bd596781348595e3a84c03cfeaeb2f5"
server_key_version: "encryptionkey@0"
I20260925 04:37:50.727269 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "968de3ef59c3489c8a717de1b3de8bec"
format_stamp: "Formatted at 2026-09-25 04:37:50 on dist-test-slave-r8r3"
server_key: "1fdbe4824962e94c8fcde0ebcba779c2"
server_key_iv: "5bd596781348595e3a84c03cfeaeb2f5"
server_key_version: "encryptionkey@0"
I20260925 04:37:50.731451 17316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:37:50.733901 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:50.735152 17316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:50.735255 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "968de3ef59c3489c8a717de1b3de8bec"
format_stamp: "Formatted at 2026-09-25 04:37:50 on dist-test-slave-r8r3"
server_key: "1fdbe4824962e94c8fcde0ebcba779c2"
server_key_iv: "5bd596781348595e3a84c03cfeaeb2f5"
server_key_version: "encryptionkey@0"
I20260925 04:37:50.735342 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:50.740005 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:50.742976 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:50.743148 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:50.750631 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41177
I20260925 04:37:50.750674 17383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41177 every 8 connection(s)
I20260925 04:37:50.751566 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:50.754354 17384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:50.757002 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17316
I20260925 04:37:50.757156 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:50.757467 27835 external_mini_cluster.cc:1428] Setting key 35f1cea86348c366a5e7cac1e18d53e8
I20260925 04:37:50.760864 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec: Bootstrap starting.
I20260925 04:37:50.762982 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:50.759043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:51643 (local address 127.0.0.1:41177)
0925 04:37:50.759534 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:50.759549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:50.759586 (+ 37us) server_negotiation.cc:407] Connection header received
0925 04:37:50.760245 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:50.760264 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:50.760554 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:50.760927 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:50.761772 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:50.761776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:50.761789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:50.761800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:50.761855 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:50.761874 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:50.762027 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:50.762108 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 04:37:50.762272 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":145,"threads_started":1}
I20260925 04:37:50.763703 17384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:50.764505 17384 log.cc:824] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:50.765851 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:50.766708 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec: No bootstrap required, opened a new log
I20260925 04:37:50.769724 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:50.767921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:49335 (local address 127.0.0.1:41177)
0925 04:37:50.768036 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:50.768046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:50.768272 (+ 226us) server_negotiation.cc:407] Connection header received
0925 04:37:50.768430 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:50.768432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:50.768467 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:50.768554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:50.769308 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:50.769311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:50.769314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:50.769326 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:50.769352 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:50.769369 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:50.769460 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:50.769556 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 04:37:50.769601 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:37:50.770907 17384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260925 04:37:50.771136 17384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:50.771210 17384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968de3ef59c3489c8a717de1b3de8bec, State: Initialized, Role: FOLLOWER
I20260925 04:37:50.771653 17384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260925 04:37:50.771798 17384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:50.771901 17384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:50.772015 17384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:50.772971 17384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260925 04:37:50.773348 17384 leader_election.cc:302] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 968de3ef59c3489c8a717de1b3de8bec; no voters:
I20260925 04:37:50.773732 17384 leader_election.cc:288] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:50.773808 17395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:50.774016 17395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [term 1 LEADER]: Becoming Leader. State: Replica: 968de3ef59c3489c8a717de1b3de8bec, State: Running, Role: LEADER
I20260925 04:37:50.774322 17395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260925 04:37:50.775086 17384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:50.776299 17397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 968de3ef59c3489c8a717de1b3de8bec. Latest consensus state: current_term: 1 leader_uuid: "968de3ef59c3489c8a717de1b3de8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260925 04:37:50.776340 17396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "968de3ef59c3489c8a717de1b3de8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968de3ef59c3489c8a717de1b3de8bec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260925 04:37:50.776508 17396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:50.776506 17397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:50.776877 17404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:50.780283 17404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:50.786332 17404 catalog_manager.cc:1347] Generated new cluster ID: da6c1381b82b4920867e160e560deb94
I20260925 04:37:50.786427 17404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:50.799460 17404 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:50.799829 17404 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:50.808019 17404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 968de3ef59c3489c8a717de1b3de8bec: Generated new TSK 0
I20260925 04:37:50.808902 17404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:50.815241 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17316
2026-09-25T04:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:50 dist-test-slave-r8r3 krb5kdc[17416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:50 dist-test-slave-r8r3 krb5kdc[17416](info): set up 2 sockets
Sep 25 04:37:50 dist-test-slave-r8r3 krb5kdc[17416](info): commencing operation
krb5kdc: starting...
W20260925 04:37:51.846271 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:51 dist-test-slave-r8r3 krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:51.972774 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:36785
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:52.084745 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:52.085073 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:52.085160 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:52.089079 17430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:52.089179 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:52.089222 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:52.089278 17430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:52.089339 17430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:52.095417 17430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:36785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:52.096628 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:52.097797 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:52.106707 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:52.106658 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:52.106741 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:52.106961 17430 server_base.cc:1034] running on GCE node
I20260925 04:37:52.107621 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:52.108845 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:52.110023 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790311072109992 us; error 53 us; skew 500 ppm
Sep 25 04:37:52 dist-test-slave-r8r3 krb5kdc[17416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311072, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:52.113560 17430 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:52.114500 17430 webserver.cc:460] Webserver started at http://127.27.46.254:44341/ using document root <none> and password file <none>
I20260925 04:37:52.115106 17430 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:52.115202 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:52.115444 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:52.117604 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "72d2ba495d7442d8b693662962ead00b"
format_stamp: "Formatted at 2026-09-25 04:37:52 on dist-test-slave-r8r3"
server_key: "af62deaf893efbdfbc6720ff71e95799"
server_key_iv: "cc53d72a1599d34de34dcf9933817bfb"
server_key_version: "encryptionkey@0"
I20260925 04:37:52.118188 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "72d2ba495d7442d8b693662962ead00b"
format_stamp: "Formatted at 2026-09-25 04:37:52 on dist-test-slave-r8r3"
server_key: "af62deaf893efbdfbc6720ff71e95799"
server_key_iv: "cc53d72a1599d34de34dcf9933817bfb"
server_key_version: "encryptionkey@0"
I20260925 04:37:52.122059 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:37:52.124626 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:52.125922 17430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:52.126071 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "72d2ba495d7442d8b693662962ead00b"
format_stamp: "Formatted at 2026-09-25 04:37:52 on dist-test-slave-r8r3"
server_key: "af62deaf893efbdfbc6720ff71e95799"
server_key_iv: "cc53d72a1599d34de34dcf9933817bfb"
server_key_version: "encryptionkey@0"
I20260925 04:37:52.126202 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:52.142041 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:52.145164 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:52.145390 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:52.152649 17430 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:36785
I20260925 04:37:52.152676 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:36785 every 8 connection(s)
I20260925 04:37:52.153699 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:52.156611 17501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:52.159919 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17430
I20260925 04:37:52.160034 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:52.160326 27835 external_mini_cluster.cc:1428] Setting key 8548f485a314d1f5964d0ad55bc37db3
I20260925 04:37:52.162974 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b: Bootstrap starting.
I20260925 04:37:52.165267 17501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:52.166138 17501 log.cc:824] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:52.168426 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b: No bootstrap required, opened a new log
Sep 25 04:37:52 dist-test-slave-r8r3 krb5kdc[17416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:52.172781 17501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } }
I20260925 04:37:52.173077 17501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:52.173204 17501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d2ba495d7442d8b693662962ead00b, State: Initialized, Role: FOLLOWER
I20260925 04:37:52.173782 17501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } }
I20260925 04:37:52.173959 17501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:52.174031 17501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:52.174132 17501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:52.175366 17501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } }
I20260925 04:37:52.175771 17501 leader_election.cc:302] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d2ba495d7442d8b693662962ead00b; no voters:
I20260925 04:37:52.176128 17501 leader_election.cc:288] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:52.176257 17506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:52.176518 17506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [term 1 LEADER]: Becoming Leader. State: Replica: 72d2ba495d7442d8b693662962ead00b, State: Running, Role: LEADER
I20260925 04:37:52.176543 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:52.161909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57411 (local address 127.27.46.254:36785)
0925 04:37:52.162366 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:52.162383 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:52.162418 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:37:52.163041 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:52.163056 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:52.163478 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:52.163816 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:52.164651 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:52.165771 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:52.166746 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:52.167383 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:52.167547 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:52.170343 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:52.170350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:52.170370 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:52.170408 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:52.172998 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:52.173642 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:52.173648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:52.173655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:52.173795 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:52.174195 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:52.174199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:52.174202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:52.174730 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:52.174957 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:52.175211 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 04:37:52.175470 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":146,"threads_started":1}
I20260925 04:37:52.177238 17506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } }
I20260925 04:37:52.177428 17501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:52.178890 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d2ba495d7442d8b693662962ead00b. Latest consensus state: current_term: 1 leader_uuid: "72d2ba495d7442d8b693662962ead00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } } }
I20260925 04:37:52.179050 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:52.179569 17518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:52.180279 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d2ba495d7442d8b693662962ead00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d2ba495d7442d8b693662962ead00b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 36785 } } }
I20260925 04:37:52.180369 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:52.180426 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:52.182904 17518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:52.189445 17518 catalog_manager.cc:1347] Generated new cluster ID: d7061139da46403d84fa2de941df995a
I20260925 04:37:52.189590 17518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:52.191342 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:52.182576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57123 (local address 127.27.46.254:36785)
0925 04:37:52.182689 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:52.182694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:52.182964 (+ 270us) server_negotiation.cc:407] Connection header received
0925 04:37:52.183087 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:52.183090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:52.183137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:52.183222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:52.183933 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:52.184479 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:52.185531 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:52.185896 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:52.186009 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:52.187612 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:52.187616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:52.187617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:52.187640 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:52.189266 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:52.189892 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:52.189896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:52.189898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:52.189991 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:52.190396 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:52.190400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:52.190402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:52.190638 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:52.190816 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:52.191076 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 04:37:52.191182 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 04:37:52.199831 17518 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:52.200799 17518 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:52.205295 17518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d2ba495d7442d8b693662962ead00b: Generated new TSK 0
I20260925 04:37:52.206061 17518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:52.212345 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17430
W20260925 04:37:52.218357 17414 connection.cc:535] client connection to 127.27.46.254:36785 recv error: Network error: failed to read from TLS socket (remote: 127.27.46.254:36785): Connection reset by peer (error 104)
2026-09-25T04:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:52 dist-test-slave-r8r3 krb5kdc[17533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:52 dist-test-slave-r8r3 krb5kdc[17533](info): set up 2 sockets
Sep 25 04:37:52 dist-test-slave-r8r3 krb5kdc[17533](info): commencing operation
krb5kdc: starting...
W20260925 04:37:53.249300 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:53 dist-test-slave-r8r3 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:53.372421 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35475
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:46011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:53.483690 17546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:53.483994 17546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:53.484037 17546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:53.488078 17546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:53.488148 17546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:53.488168 17546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:53.488185 17546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:53.488200 17546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:53.493453 17546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:46011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:53.494597 17546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:53.495608 17546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:37:53.504196 17546 server_base.cc:1034] running on GCE node
W20260925 04:37:53.504144 17551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:53.504139 17552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:53.504144 17554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:53.504911 17546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:53.506138 17546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:53.507320 17546 hybrid_clock.cc:648] HybridClock initialized: now 1790311073507302 us; error 30 us; skew 500 ppm
Sep 25 04:37:53 dist-test-slave-r8r3 krb5kdc[17533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:53.510098 17546 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:53.510960 17546 webserver.cc:460] Webserver started at http://127.27.46.254:37601/ using document root <none> and password file <none>
I20260925 04:37:53.511478 17546 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:53.511533 17546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:53.511693 17546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:53.513671 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "70a1cda3fbea41458a7d87313663e132"
format_stamp: "Formatted at 2026-09-25 04:37:53 on dist-test-slave-r8r3"
server_key: "58cf210165101488c04280f83b980367"
server_key_iv: "c159981e6202ebce19546538eafb6667"
server_key_version: "encryptionkey@0"
I20260925 04:37:53.514186 17546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "70a1cda3fbea41458a7d87313663e132"
format_stamp: "Formatted at 2026-09-25 04:37:53 on dist-test-slave-r8r3"
server_key: "58cf210165101488c04280f83b980367"
server_key_iv: "c159981e6202ebce19546538eafb6667"
server_key_version: "encryptionkey@0"
I20260925 04:37:53.517861 17546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 04:37:53.520172 17561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:53.521379 17546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:37:53.521562 17546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "70a1cda3fbea41458a7d87313663e132"
format_stamp: "Formatted at 2026-09-25 04:37:53 on dist-test-slave-r8r3"
server_key: "58cf210165101488c04280f83b980367"
server_key_iv: "c159981e6202ebce19546538eafb6667"
server_key_version: "encryptionkey@0"
I20260925 04:37:53.521674 17546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:53.547915 17546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:53.551291 17546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:53.551518 17546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:53.559101 17546 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35475
I20260925 04:37:53.559176 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35475 every 8 connection(s)
I20260925 04:37:53.560092 17546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:53.563077 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:53.568457 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: Bootstrap starting.
I20260925 04:37:53.569541 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17546
I20260925 04:37:53.569703 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:53.570011 27835 external_mini_cluster.cc:1428] Setting key 72e50b2b4f3a3ea2ea68aad211b2294d
I20260925 04:37:53.571518 17616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:53.572256 17616 log.cc:824] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:53.574222 17616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: No bootstrap required, opened a new log
I20260925 04:37:53.577545 17616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } }
I20260925 04:37:53.577802 17616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:53.577883 17616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a1cda3fbea41458a7d87313663e132, State: Initialized, Role: FOLLOWER
I20260925 04:37:53.578351 17616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } }
I20260925 04:37:53.578524 17616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:53.578601 17616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:53.578691 17616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:53.579171 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:53.571161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41803 (local address 127.27.46.254:35475)
0925 04:37:53.571656 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:53.571672 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:53.571706 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:37:53.572397 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:53.572416 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:53.572832 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:53.573226 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:53.573679 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:53.574613 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:53.575539 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:53.575995 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:53.576158 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:53.577121 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:53.577126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:53.577138 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:53.577582 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:53.577638 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:53.577662 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:53.577807 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:53.578008 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 04:37:53.578147 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":140,"threads_started":1}
I20260925 04:37:53.579857 17616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } }
I20260925 04:37:53.580207 17616 leader_election.cc:302] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70a1cda3fbea41458a7d87313663e132; no voters:
I20260925 04:37:53.580622 17616 leader_election.cc:288] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:53.580683 17621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:53.580855 17621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [term 1 LEADER]: Becoming Leader. State: Replica: 70a1cda3fbea41458a7d87313663e132, State: Running, Role: LEADER
I20260925 04:37:53.581634 17621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } }
I20260925 04:37:53.581892 17616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:53.582472 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:53.583423 17622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70a1cda3fbea41458a7d87313663e132. Latest consensus state: current_term: 1 leader_uuid: "70a1cda3fbea41458a7d87313663e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } } }
I20260925 04:37:53.583561 17622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:53.583847 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70a1cda3fbea41458a7d87313663e132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a1cda3fbea41458a7d87313663e132" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35475 } } }
I20260925 04:37:53.583942 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132 [sys.catalog]: This master's current role is: LEADER
W20260925 04:37:53.587184 17640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:37:53.587267 17640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:37:53.587373 17633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:53.588037 17633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:53.591087 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:53.586221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41435 (local address 127.27.46.254:35475)
0925 04:37:53.586337 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:53.586342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:53.586657 (+ 315us) server_negotiation.cc:407] Connection header received
0925 04:37:53.586784 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:53.586787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:53.586849 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:53.586930 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:53.587270 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:53.588014 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:53.589151 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:53.589558 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:53.589667 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:53.590547 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:53.590551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:53.590554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:53.590596 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:53.590620 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:53.590649 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:53.590765 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:53.590897 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:37:53.590931 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:37:53.594798 17633 catalog_manager.cc:1347] Generated new cluster ID: 40191418a8704757b384484bd8f738d8
I20260925 04:37:53.594867 17633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:53.612524 17633 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:53.613462 17633 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:53.629268 17633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70a1cda3fbea41458a7d87313663e132: Generated new TSK 0
I20260925 04:37:53.630060 17633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:53.640951 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17546
W20260925 04:37:53.647156 17531 connection.cc:535] client connection to 127.27.46.254:35475 recv error: Network error: failed to read from TLS socket (remote: 127.27.46.254:35475): Connection reset by peer (error 104)
2026-09-25T04:37:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:53 dist-test-slave-r8r3 krb5kdc[17647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:53 dist-test-slave-r8r3 krb5kdc[17647](info): set up 2 sockets
Sep 25 04:37:53 dist-test-slave-r8r3 krb5kdc[17647](info): commencing operation
krb5kdc: starting...
W20260925 04:37:54.669112 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:54 dist-test-slave-r8r3 krb5kdc[17647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:54.792657 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43559
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:43559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:54.905112 17660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:54.905440 17660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:54.905556 17660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:54.909659 17660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:54.909767 17660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:54.909806 17660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:54.909865 17660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:54.909905 17660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:54.915299 17660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:43559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:43559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:54.916438 17660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:54.917527 17660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:54.926821 17666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:54.926822 17665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:54.926822 17668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:54.927210 17660 server_base.cc:1034] running on GCE node
I20260925 04:37:54.927845 17660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:54.929062 17660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:54.930279 17660 hybrid_clock.cc:648] HybridClock initialized: now 1790311074930270 us; error 36 us; skew 500 ppm
Sep 25 04:37:54 dist-test-slave-r8r3 krb5kdc[17647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:54.933187 17660 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:54.934185 17660 webserver.cc:460] Webserver started at http://127.27.46.254:38561/ using document root <none> and password file <none>
I20260925 04:37:54.934803 17660 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:54.934901 17660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:54.935142 17660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:54.937168 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "4c94090ee88143e489a78e796379b2b8"
format_stamp: "Formatted at 2026-09-25 04:37:54 on dist-test-slave-r8r3"
server_key: "e82e63915b30e2c2e60eec8560229134"
server_key_iv: "d25f8096ee12baa6f41fd8950140501b"
server_key_version: "encryptionkey@0"
I20260925 04:37:54.937765 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "4c94090ee88143e489a78e796379b2b8"
format_stamp: "Formatted at 2026-09-25 04:37:54 on dist-test-slave-r8r3"
server_key: "e82e63915b30e2c2e60eec8560229134"
server_key_iv: "d25f8096ee12baa6f41fd8950140501b"
server_key_version: "encryptionkey@0"
I20260925 04:37:54.942061 17660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:37:54.944648 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:54.945973 17660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:54.946120 17660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "4c94090ee88143e489a78e796379b2b8"
format_stamp: "Formatted at 2026-09-25 04:37:54 on dist-test-slave-r8r3"
server_key: "e82e63915b30e2c2e60eec8560229134"
server_key_iv: "d25f8096ee12baa6f41fd8950140501b"
server_key_version: "encryptionkey@0"
I20260925 04:37:54.946242 17660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:54.951012 17660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:54.954312 17660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:54.954519 17660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:54.961784 17660 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:43559
I20260925 04:37:54.961838 17731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:43559 every 8 connection(s)
I20260925 04:37:54.962828 17660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:54.965736 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:54.969527 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17660
I20260925 04:37:54.969682 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:54.970000 27835 external_mini_cluster.cc:1428] Setting key c20449bb711ac8e8cc24c6af4a08bb1e
I20260925 04:37:54.972045 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8: Bootstrap starting.
I20260925 04:37:54.974438 17732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:54.975193 17732 log.cc:824] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:54.975358 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:54.971549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48197 (local address 127.27.46.254:43559)
0925 04:37:54.972036 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:54.972052 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:54.972088 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:37:54.972777 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:54.972793 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:54.973067 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:54.973409 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:54.974181 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:54.974185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:54.974196 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:54.974205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:54.974260 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:54.974278 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:54.974385 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:54.974480 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 04:37:54.974636 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":145,"threads_started":1}
I20260925 04:37:54.977378 17732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8: No bootstrap required, opened a new log
I20260925 04:37:54.978223 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:54.981607 17732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } }
I20260925 04:37:54.982015 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:54.980324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39149 (local address 127.27.46.254:43559)
0925 04:37:54.980439 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:54.980443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:54.980622 (+ 179us) server_negotiation.cc:407] Connection header received
0925 04:37:54.980762 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:54.980764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:54.980794 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:54.980850 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:54.981586 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:54.981592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:54.981599 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:54.981609 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:54.981633 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:54.981643 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:54.981711 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:54.981829 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 04:37:54.981862 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:37:54.982019 17732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:54.982148 17732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c94090ee88143e489a78e796379b2b8, State: Initialized, Role: FOLLOWER
I20260925 04:37:54.982642 17732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } }
I20260925 04:37:54.982806 17732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:54.982882 17732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:54.982986 17732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:54.983984 17732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } }
I20260925 04:37:54.984355 17732 leader_election.cc:302] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c94090ee88143e489a78e796379b2b8; no voters:
I20260925 04:37:54.984684 17732 leader_election.cc:288] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:54.984817 17742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:54.985067 17742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4c94090ee88143e489a78e796379b2b8, State: Running, Role: LEADER
I20260925 04:37:54.985412 17742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } }
I20260925 04:37:54.986068 17732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:54.987160 17744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c94090ee88143e489a78e796379b2b8. Latest consensus state: current_term: 1 leader_uuid: "4c94090ee88143e489a78e796379b2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } } }
I20260925 04:37:54.987329 17744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:54.987720 17751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:54.988680 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c94090ee88143e489a78e796379b2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c94090ee88143e489a78e796379b2b8" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 43559 } } }
I20260925 04:37:54.988788 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:54.991007 17751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:54.996850 17751 catalog_manager.cc:1347] Generated new cluster ID: 453750cc5dcf4ff3814f575308c9f3d6
I20260925 04:37:54.996937 17751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:55.008546 17751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:55.008888 17751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:55.016614 17751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c94090ee88143e489a78e796379b2b8: Generated new TSK 0
I20260925 04:37:55.017620 17751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:55.029559 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17660
2026-09-25T04:37:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:55 dist-test-slave-r8r3 krb5kdc[17763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:55 dist-test-slave-r8r3 krb5kdc[17763](info): set up 2 sockets
Sep 25 04:37:55 dist-test-slave-r8r3 krb5kdc[17763](info): commencing operation
krb5kdc: starting...
W20260925 04:37:56.078138 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:56.206354 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34331
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:34331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:56.324467 17776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:56.324750 17776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:56.324811 17776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:56.328949 17776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:56.329015 17776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:56.329034 17776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:56.329051 17776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:56.329066 17776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:56.334424 17776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:34331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:56.335584 17776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:56.336680 17776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:56.345655 17782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:56.345655 17781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:56.345824 17776 server_base.cc:1034] running on GCE node
W20260925 04:37:56.345630 17784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:56.346539 17776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:56.347683 17776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:56.348893 17776 hybrid_clock.cc:648] HybridClock initialized: now 1790311076348882 us; error 25 us; skew 500 ppm
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311076, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:56.356994 17776 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:56.358114 17776 webserver.cc:460] Webserver started at http://127.27.46.254:39447/ using document root <none> and password file <none>
I20260925 04:37:56.358714 17776 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:56.358824 17776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:56.359076 17776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:56.361198 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "c63f0f9206224042aee8261abfb0e82b"
format_stamp: "Formatted at 2026-09-25 04:37:56 on dist-test-slave-r8r3"
server_key: "69b85ee77ab29076228f843266690f62"
server_key_iv: "dc057cbbf30283244ba6f5316bc2fa72"
server_key_version: "encryptionkey@0"
I20260925 04:37:56.361830 17776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "c63f0f9206224042aee8261abfb0e82b"
format_stamp: "Formatted at 2026-09-25 04:37:56 on dist-test-slave-r8r3"
server_key: "69b85ee77ab29076228f843266690f62"
server_key_iv: "dc057cbbf30283244ba6f5316bc2fa72"
server_key_version: "encryptionkey@0"
I20260925 04:37:56.365707 17776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:37:56.368263 17791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:56.369650 17776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:37:56.369814 17776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "c63f0f9206224042aee8261abfb0e82b"
format_stamp: "Formatted at 2026-09-25 04:37:56 on dist-test-slave-r8r3"
server_key: "69b85ee77ab29076228f843266690f62"
server_key_iv: "dc057cbbf30283244ba6f5316bc2fa72"
server_key_version: "encryptionkey@0"
I20260925 04:37:56.369942 17776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:56.384588 17776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:56.387534 17776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:56.387779 17776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:56.395431 17776 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:34331
I20260925 04:37:56.395457 17848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:34331 every 8 connection(s)
I20260925 04:37:56.396437 17776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:56.399343 17849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:56.403276 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17776
I20260925 04:37:56.403452 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:56.403765 27835 external_mini_cluster.cc:1428] Setting key 439274cd5098ba5c08a5ae184c432548
I20260925 04:37:56.405755 17849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b: Bootstrap starting.
I20260925 04:37:56.408480 17849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:56.409454 17849 log.cc:824] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:56.411649 17849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b: No bootstrap required, opened a new log
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:37:56.416097 17849 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } }
I20260925 04:37:56.416406 17849 raft_consensus.cc:374] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:56.416463 17849 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63f0f9206224042aee8261abfb0e82b, State: Initialized, Role: FOLLOWER
I20260925 04:37:56.416975 17849 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } }
I20260925 04:37:56.417160 17849 raft_consensus.cc:388] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:56.417240 17849 raft_consensus.cc:482] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:56.417352 17849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:56.418659 17849 raft_consensus.cc:504] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } }
I20260925 04:37:56.419051 17849 leader_election.cc:302] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c63f0f9206224042aee8261abfb0e82b; no voters:
I20260925 04:37:56.419400 17849 leader_election.cc:288] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:56.419524 17854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:56.419811 17854 raft_consensus.cc:686] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [term 1 LEADER]: Becoming Leader. State: Replica: c63f0f9206224042aee8261abfb0e82b, State: Running, Role: LEADER
I20260925 04:37:56.420150 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:56.405279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39187 (local address 127.27.46.254:34331)
0925 04:37:56.405731 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:56.405755 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:56.405791 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:37:56.406472 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:56.406488 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:56.406875 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:56.407313 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:56.408638 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:56.410135 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:56.411157 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:56.411850 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:56.412022 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:56.414589 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:56.414596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:56.414615 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:56.414652 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:56.416675 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:56.417141 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:56.417147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:56.417153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:56.417304 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:56.417700 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:56.417703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:56.417705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:56.418233 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:56.418473 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:56.418748 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:37:56.419075 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":155,"threads_started":1}
I20260925 04:37:56.420181 17854 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } }
I20260925 04:37:56.420768 17849 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:56.421468 17854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63f0f9206224042aee8261abfb0e82b. Latest consensus state: current_term: 1 leader_uuid: "c63f0f9206224042aee8261abfb0e82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } } }
I20260925 04:37:56.421617 17854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:56.422050 17861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:56.423578 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:56.425625 17861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:56.425592 17856 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c63f0f9206224042aee8261abfb0e82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63f0f9206224042aee8261abfb0e82b" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 34331 } } }
I20260925 04:37:56.425799 17856 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:56.431614 17861 catalog_manager.cc:1347] Generated new cluster ID: dc16b26e41a648758d8b2905f4cbc828
I20260925 04:37:56.431684 17861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:56.437526 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:56.426811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55265 (local address 127.27.46.254:34331)
0925 04:37:56.426930 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:56.426934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:56.428005 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 04:37:56.428097 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:56.428101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:56.428156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:56.428266 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:56.429088 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:56.429958 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:56.431166 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:56.431612 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:56.431717 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:56.433653 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:56.433657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:56.433660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:37:56.433688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:56.435607 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:56.436109 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:56.436113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:56.436115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:56.436213 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:37:56.436514 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:37:56.436517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:37:56.436519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:37:56.436802 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:56.436956 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:56.437196 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 04:37:56.437327 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":47}
I20260925 04:37:56.442994 17861 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:56.444190 17861 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:56.452937 17861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c63f0f9206224042aee8261abfb0e82b: Generated new TSK 0
I20260925 04:37:56.453814 17861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:56.459462 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17776
2026-09-25T04:37:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17880](info): set up 2 sockets
Sep 25 04:37:56 dist-test-slave-r8r3 krb5kdc[17880](info): commencing operation
krb5kdc: starting...
W20260925 04:37:57.506108 27835 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:57 dist-test-slave-r8r3 krb5kdc[17880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:57.629963 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46245
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:46245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:57.734678 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:57.734908 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:57.734944 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:57.738677 17894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:57.738731 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:57.738751 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:57.738767 17894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:57.738790 17894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:57.743841 17894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:46245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:46245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:57.744854 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:57.745915 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:57.754606 17902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:57.754624 17899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:57.754722 17894 server_base.cc:1034] running on GCE node
W20260925 04:37:57.754618 17900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:57.755549 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:57.756650 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:57.757840 17894 hybrid_clock.cc:648] HybridClock initialized: now 1790311077757816 us; error 44 us; skew 500 ppm
Sep 25 04:37:57 dist-test-slave-r8r3 krb5kdc[17880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311077, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:57.760882 17894 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:57.761775 17894 webserver.cc:460] Webserver started at http://127.27.46.254:43293/ using document root <none> and password file <none>
I20260925 04:37:57.762292 17894 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:57.762353 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:57.762529 17894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:57.764492 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "2cfea23558bd43638ec3c5e424cab4c9"
format_stamp: "Formatted at 2026-09-25 04:37:57 on dist-test-slave-r8r3"
server_key: "b2e0ba483d7f44a6c1b8a535d2dcee3e"
server_key_iv: "ac022ccb63962148ed804d9769c8606d"
server_key_version: "encryptionkey@0"
I20260925 04:37:57.764989 17894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "2cfea23558bd43638ec3c5e424cab4c9"
format_stamp: "Formatted at 2026-09-25 04:37:57 on dist-test-slave-r8r3"
server_key: "b2e0ba483d7f44a6c1b8a535d2dcee3e"
server_key_iv: "ac022ccb63962148ed804d9769c8606d"
server_key_version: "encryptionkey@0"
I20260925 04:37:57.768494 17894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 04:37:57.770759 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:57.771909 17894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 04:37:57.772046 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "2cfea23558bd43638ec3c5e424cab4c9"
format_stamp: "Formatted at 2026-09-25 04:37:57 on dist-test-slave-r8r3"
server_key: "b2e0ba483d7f44a6c1b8a535d2dcee3e"
server_key_iv: "ac022ccb63962148ed804d9769c8606d"
server_key_version: "encryptionkey@0"
I20260925 04:37:57.772153 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:57.783092 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:57.786127 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:57.786314 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:57.792793 17894 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:46245
I20260925 04:37:57.792865 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:46245 every 8 connection(s)
I20260925 04:37:57.793740 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:57.795708 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 17894
I20260925 04:37:57.795866 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:57.796159 27835 external_mini_cluster.cc:1428] Setting key 98ca906217556e8ceb928f1ff8f6c414
I20260925 04:37:57.796468 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:57.803314 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9: Bootstrap starting.
I20260925 04:37:57.805544 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:57.797447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47625 (local address 127.27.46.254:46245)
0925 04:37:57.797938 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:57.797956 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:57.797998 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:37:57.798748 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:57.798769 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:57.799190 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:57.799522 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:57.799953 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:57.801398 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:57.802221 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:57.802819 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:57.802960 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:57.803706 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:57.803713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:57.803728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:57.804194 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:57.804260 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:57.804291 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:57.804389 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:57.804498 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 04:37:57.804726 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":119,"threads_started":1}
I20260925 04:37:57.805907 17962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:57.806602 17962 log.cc:824] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:57.808079 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:57.808449 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9: No bootstrap required, opened a new log
I20260925 04:37:57.812331 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } }
I20260925 04:37:57.812599 17962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:57.812687 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfea23558bd43638ec3c5e424cab4c9, State: Initialized, Role: FOLLOWER
I20260925 04:37:57.813127 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } }
I20260925 04:37:57.813297 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:57.813361 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:57.813463 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:57.813853 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:57.810007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39779 (local address 127.27.46.254:46245)
0925 04:37:57.810116 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:57.810120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:57.810265 (+ 145us) server_negotiation.cc:407] Connection header received
0925 04:37:57.810366 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:57.810368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:57.810403 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:57.810467 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:57.810755 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:57.811201 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:57.812159 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:37:57.812362 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:37:57.812430 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:37:57.813312 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:57.813315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:57.813317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:57.813329 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:57.813359 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:57.813391 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:57.813458 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:57.813647 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 04:37:57.813706 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:37:57.814436 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } }
I20260925 04:37:57.814816 17962 leader_election.cc:302] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cfea23558bd43638ec3c5e424cab4c9; no voters:
I20260925 04:37:57.815121 17962 leader_election.cc:288] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:57.815188 17972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:57.815395 17972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [term 1 LEADER]: Becoming Leader. State: Replica: 2cfea23558bd43638ec3c5e424cab4c9, State: Running, Role: LEADER
I20260925 04:37:57.815697 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } }
I20260925 04:37:57.816277 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:57.817412 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cfea23558bd43638ec3c5e424cab4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } } }
I20260925 04:37:57.817556 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:57.817651 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cfea23558bd43638ec3c5e424cab4c9. Latest consensus state: current_term: 1 leader_uuid: "2cfea23558bd43638ec3c5e424cab4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cfea23558bd43638ec3c5e424cab4c9" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 46245 } } }
I20260925 04:37:57.817742 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:57.818598 17981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:57.821233 17981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:57.826787 17981 catalog_manager.cc:1347] Generated new cluster ID: 11cd2b2ec8df42b8874245d197a59ddc
I20260925 04:37:57.826849 17981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:57.834774 17981 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:57.835922 17981 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:57.842304 17981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cfea23558bd43638ec3c5e424cab4c9: Generated new TSK 0
I20260925 04:37:57.843046 17981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:57.846894 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 17894
2026-09-25T04:37:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:57 dist-test-slave-r8r3 krb5kdc[17993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:57 dist-test-slave-r8r3 krb5kdc[17993](info): set up 2 sockets
Sep 25 04:37:57 dist-test-slave-r8r3 krb5kdc[17993](info): commencing operation
krb5kdc: starting...
W20260925 04:37:58.856613 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:37:58 dist-test-slave-r8r3 krb5kdc[17993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:37:58.976585 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33845
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:37:59.084617 18007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:37:59.084853 18007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:37:59.084888 18007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:37:59.088361 18007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:37:59.088408 18007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:37:59.088424 18007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:37:59.088438 18007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:37:59.088450 18007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:37:59.093147 18007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:37:59.094159 18007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:37:59.095160 18007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:37:59.103394 18015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:59.103431 18012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:37:59.103463 18013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:37:59.103479 18007 server_base.cc:1034] running on GCE node
I20260925 04:37:59.104218 18007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:37:59.105261 18007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:37:59.106455 18007 hybrid_clock.cc:648] HybridClock initialized: now 1790311079106438 us; error 25 us; skew 500 ppm
Sep 25 04:37:59 dist-test-slave-r8r3 krb5kdc[17993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311079, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:37:59.109102 18007 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:37:59.109984 18007 webserver.cc:460] Webserver started at http://127.27.46.254:45295/ using document root <none> and password file <none>
I20260925 04:37:59.110471 18007 fs_manager.cc:359] Metadata directory not provided
I20260925 04:37:59.110524 18007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:37:59.110677 18007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:37:59.112537 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "ea9e4134f3af4135a247a30048cda707"
format_stamp: "Formatted at 2026-09-25 04:37:59 on dist-test-slave-r8r3"
server_key: "42bb89faef83220acad4ef486ab54775"
server_key_iv: "4095efe3cf0bdcf8e6d802940c91085a"
server_key_version: "encryptionkey@0"
I20260925 04:37:59.113025 18007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "ea9e4134f3af4135a247a30048cda707"
format_stamp: "Formatted at 2026-09-25 04:37:59 on dist-test-slave-r8r3"
server_key: "42bb89faef83220acad4ef486ab54775"
server_key_iv: "4095efe3cf0bdcf8e6d802940c91085a"
server_key_version: "encryptionkey@0"
I20260925 04:37:59.116339 18007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 04:37:59.118564 18022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:37:59.119828 18007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:37:59.119971 18007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "ea9e4134f3af4135a247a30048cda707"
format_stamp: "Formatted at 2026-09-25 04:37:59 on dist-test-slave-r8r3"
server_key: "42bb89faef83220acad4ef486ab54775"
server_key_iv: "4095efe3cf0bdcf8e6d802940c91085a"
server_key_version: "encryptionkey@0"
I20260925 04:37:59.120086 18007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:37:59.124557 18007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:37:59.127751 18007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:37:59.127956 18007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:37:59.134572 18007 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:33845
I20260925 04:37:59.134598 18075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:33845 every 8 connection(s)
I20260925 04:37:59.135388 18007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:37:59.138448 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:37:59.142447 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18007
I20260925 04:37:59.142593 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:37:59.142861 27835 external_mini_cluster.cc:1428] Setting key 6891a3d0c5a90820e0fec562409f6d5f
I20260925 04:37:59.144611 18077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707: Bootstrap starting.
I20260925 04:37:59.147128 18077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707: Neither blocks nor log segments found. Creating new log.
I20260925 04:37:59.147953 18077 log.cc:824] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707: Log is configured to *not* fsync() on all Append() calls
I20260925 04:37:59.148129 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:59.144247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57039 (local address 127.27.46.254:33845)
0925 04:37:59.144692 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:59.144711 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:59.144749 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:37:59.145519 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:59.145542 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:59.145902 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:59.146225 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:59.146870 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:59.146875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:59.146888 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:59.146898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:59.146954 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:59.146979 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:59.147085 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:59.147160 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:37:59.147294 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":118,"threads_started":1}
I20260925 04:37:59.150091 18077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707: No bootstrap required, opened a new log
I20260925 04:37:59.150960 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:37:59.154134 18077 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } }
I20260925 04:37:59.154434 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:37:59.152986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50053 (local address 127.27.46.254:33845)
0925 04:37:59.153118 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:37:59.153123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:37:59.153191 (+ 68us) server_negotiation.cc:407] Connection header received
0925 04:37:59.153280 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:37:59.153283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:37:59.153313 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:37:59.153393 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:37:59.154079 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:37:59.154082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:37:59.154084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:37:59.154095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:37:59.154121 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:37:59.154134 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:37:59.154245 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:37:59.154297 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 04:37:59.154333 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:37:59.154441 18077 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:37:59.154582 18077 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9e4134f3af4135a247a30048cda707, State: Initialized, Role: FOLLOWER
I20260925 04:37:59.155004 18077 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } }
I20260925 04:37:59.155165 18077 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:37:59.155243 18077 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:37:59.155380 18077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:37:59.156207 18077 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } }
I20260925 04:37:59.156558 18077 leader_election.cc:302] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9e4134f3af4135a247a30048cda707; no voters:
I20260925 04:37:59.156836 18077 leader_election.cc:288] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:37:59.156951 18088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:37:59.157186 18088 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [term 1 LEADER]: Becoming Leader. State: Replica: ea9e4134f3af4135a247a30048cda707, State: Running, Role: LEADER
I20260925 04:37:59.157454 18088 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } }
I20260925 04:37:59.157965 18077 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:37:59.158926 18090 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9e4134f3af4135a247a30048cda707. Latest consensus state: current_term: 1 leader_uuid: "ea9e4134f3af4135a247a30048cda707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } } }
I20260925 04:37:59.159049 18090 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:59.159405 18097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:37:59.159452 18089 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9e4134f3af4135a247a30048cda707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e4134f3af4135a247a30048cda707" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 33845 } } }
I20260925 04:37:59.159639 18089 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707 [sys.catalog]: This master's current role is: LEADER
I20260925 04:37:59.162300 18097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:37:59.168080 18097 catalog_manager.cc:1347] Generated new cluster ID: 3019c42d7347461c9a3ab0ed4dd8f72e
I20260925 04:37:59.168147 18097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:37:59.183140 18097 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:37:59.183467 18097 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:37:59.192395 18097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9e4134f3af4135a247a30048cda707: Generated new TSK 0
I20260925 04:37:59.193045 18097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:37:59.200160 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18007
2026-09-25T04:37:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1353 ms)
[----------] 18 tests from AuthTokenIssuingTest (25583 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:37:59 dist-test-slave-r8r3 krb5kdc[18109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:37:59 dist-test-slave-r8r3 krb5kdc[18109](info): set up 2 sockets
Sep 25 04:37:59 dist-test-slave-r8r3 krb5kdc[18109](info): commencing operation
krb5kdc: starting...
W20260925 04:38:00.196864 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:38:00.316192 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35361
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:00.422982 18122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:00.423230 18122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:00.423252 18122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:00.423437 18122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:00.427068 18122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:00.427126 18122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:00.427143 18122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:00.427156 18122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:00.427169 18122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:00.432093 18122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:00.433040 18122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:00.434113 18122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:00.442420 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.442425 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.442451 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:00.442757 18122 server_base.cc:1034] running on GCE node
I20260925 04:38:00.443275 18122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:00.444299 18122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:00.445461 18122 hybrid_clock.cc:648] HybridClock initialized: now 1790311080445450 us; error 35 us; skew 500 ppm
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:00.447855 18122 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:38:00.448673 18122 webserver.cc:460] Webserver started at http://127.27.46.254:46423/ using document root <none> and password file <none>
I20260925 04:38:00.449177 18122 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:00.449231 18122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:00.449465 18122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:00.451491 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "33f1c29198094109b49fbf8d122d9845"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "924ad5cddc7ef1655661337769305d61"
server_key_iv: "c3b1fc0efa9adaff50295093793ff1fb"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.452077 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "33f1c29198094109b49fbf8d122d9845"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "924ad5cddc7ef1655661337769305d61"
server_key_iv: "c3b1fc0efa9adaff50295093793ff1fb"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.455770 18122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 04:38:00.458032 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:00.459174 18122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:00.459322 18122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "33f1c29198094109b49fbf8d122d9845"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "924ad5cddc7ef1655661337769305d61"
server_key_iv: "c3b1fc0efa9adaff50295093793ff1fb"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.459427 18122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:00.471295 18122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:00.473999 18122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:00.474189 18122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:00.480654 18122 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:35361
I20260925 04:38:00.480926 18193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:35361 every 8 connection(s)
I20260925 04:38:00.481690 18122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:00.482093 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18122
I20260925 04:38:00.482218 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:38:00.482506 27835 external_mini_cluster.cc:1428] Setting key b860ffe7f654db4f7c4b195d431a774b
I20260925 04:38:00.484524 18194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:00.488083 18194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.253@KRBTEST.COM" created.
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:00.503103 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.490276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.46.254:35361)
0925 04:38:00.490699 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.490713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.490931 (+ 218us) server_negotiation.cc:407] Connection header received
0925 04:38:00.491844 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.491848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.492051 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.492236 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.492917 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.493863 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.494960 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.495503 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.495636 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.498278 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.498288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.498304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.498333 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.500256 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.500747 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.500753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.500755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.500879 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.501218 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.501221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.501225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.501695 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.501812 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.502169 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.502324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"spinlock_wait_cycles":35584,"thread_start_us":159,"threads_started":1}
I20260925 04:38:00.503110 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.490277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35361 (local address 127.0.0.1:50450)
0925 04:38:00.490740 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.490789 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.491594 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.492262 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.492265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.492550 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.492770 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.492781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.494019 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.494026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.494836 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.494845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.495701 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.495708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.495873 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.496445 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.496472 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.498079 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.500376 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.500384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.500391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.500614 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.500979 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.500981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.500982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.501108 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.501807 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.501811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.501924 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.502054 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.502424 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":79,"thread_start_us":150,"threads_started":1}
I20260925 04:38:00.505579 18194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:00.506362 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.506048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39948)
0925 04:38:00.506192 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.506305 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:38:00.506583 18142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
I20260925 04:38:00.507479 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.507208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39962)
0925 04:38:00.507376 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.507424 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260925 04:38:00.507661 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:40811: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
I20260925 04:38:00.510258 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:40811
--webserver_interface=127.27.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:38:00.556995 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } attempt: 1
I20260925 04:38:00.557971 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.557680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39964)
0925 04:38:00.557834 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.557895 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 04:38:00.558717 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.558485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39966)
0925 04:38:00.558597 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.558659 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:38:00.558853 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:40811: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
W20260925 04:38:00.613176 18201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:00.613466 18201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:00.613554 18201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:00.613598 18201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:00.616948 18201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:00.616986 18201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:00.617002 18201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:00.617015 18201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:00.617028 18201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:00.621889 18201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:00.622710 18201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:00.623648 18201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:38:00.626995 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } attempt: 2
I20260925 04:38:00.627835 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.627515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39976)
0925 04:38:00.627632 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.627696 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 04:38:00.628573 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.628332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:39992)
0925 04:38:00.628450 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.628505 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:38:00.628700 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:40811: Network error: Client connection negotiation failed: client connection to 127.27.46.253:40811: connect: Connection refused (error 111)
I20260925 04:38:00.631958 18201 server_base.cc:1034] running on GCE node
W20260925 04:38:00.631973 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.631953 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.631938 18210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:00.632761 18201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:00.633862 18201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:00.635051 18201 hybrid_clock.cc:648] HybridClock initialized: now 1790311080635039 us; error 40 us; skew 500 ppm
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:00.637426 18201 init.cc:377] Logged in from keytab as kudu/127.27.46.253@KRBTEST.COM (short username kudu)
I20260925 04:38:00.638264 18201 webserver.cc:460] Webserver started at http://127.27.46.253:35671/ using document root <none> and password file <none>
I20260925 04:38:00.638921 18201 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:00.639019 18201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:00.639231 18201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:00.641273 18201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data/instance:
uuid: "85aa19e21ba646c882809d2009a42ecf"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "5963e1b78512e25a40eb080d716ceb5b"
server_key_iv: "4f96445c165d444afbbca858083682f3"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.641789 18201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal/instance:
uuid: "85aa19e21ba646c882809d2009a42ecf"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "5963e1b78512e25a40eb080d716ceb5b"
server_key_iv: "4f96445c165d444afbbca858083682f3"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.645699 18201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:38:00.648139 18217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:00.649428 18201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:00.649606 18201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "85aa19e21ba646c882809d2009a42ecf"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "5963e1b78512e25a40eb080d716ceb5b"
server_key_iv: "4f96445c165d444afbbca858083682f3"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.649734 18201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:00.673919 18201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:00.676759 18201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:00.676988 18201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:00.683871 18201 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.253:40811
I20260925 04:38:00.684175 18270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.253:40811 every 8 connection(s)
I20260925 04:38:00.684929 18201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
I20260925 04:38:00.685814 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18201
I20260925 04:38:00.685941 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-1/wal/instance
I20260925 04:38:00.686228 27835 external_mini_cluster.cc:1428] Setting key 7349cb9daf38c8706ac122275b46c171
I20260925 04:38:00.688374 18271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:00.692230 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.252@KRBTEST.COM" created.
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:00.706737 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.694219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.27.46.254:35361)
0925 04:38:00.694381 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.694385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.695357 (+ 972us) server_negotiation.cc:407] Connection header received
0925 04:38:00.696321 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.696323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.696376 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.696450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.697790 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.698640 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.699780 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.700216 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.700350 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.702930 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.702939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.702942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.702966 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.704549 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.705103 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.705106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.705110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.705189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.705566 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.705570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.705572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.705763 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.705886 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.706401 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.706528 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:38:00.707513 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.694709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35361 (local address 127.0.0.1:50464)
0925 04:38:00.695191 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.695231 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.696121 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.696613 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.696619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.697022 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.697633 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.697653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.698787 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.698789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.699642 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.699651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.700370 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.700373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.700506 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.701084 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.701100 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.702740 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.704684 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.704690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.704704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.704986 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.705298 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.705301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.705303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.705442 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.705883 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.705890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.706134 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.706329 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.706676 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260925 04:38:00.708974 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
I20260925 04:38:00.716777 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:46315
--webserver_interface=127.27.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:34223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:38:00.718747 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.709582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:40000)
0925 04:38:00.709747 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.709761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.709841 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.710480 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.710484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.710690 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.710769 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.710775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.711914 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.711921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.712530 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.712555 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.713172 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.713176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.713269 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.713750 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.713762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.714983 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.717084 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.717088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.717090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.717288 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.717660 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.717664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.717666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.717747 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.718227 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.718229 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.718316 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.718434 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.718562 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:38:00.718794 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.709739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.27.46.253:40811)
0925 04:38:00.710085 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.710096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.710117 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:38:00.710152 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.710156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.710305 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.710505 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.710905 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.711780 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.712679 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.713046 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.713178 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.715176 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.715185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.715195 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.715236 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.716978 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.717408 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.717414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.717415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.717547 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.717856 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.717859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.717861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.718104 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.718253 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.718530 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.718657 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":143,"threads_started":1}
I20260925 04:38:00.719862 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } attempt: 3
I20260925 04:38:00.720894 18271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:00.721688 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.721400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46304)
0925 04:38:00.721530 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.721619 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 04:38:00.721892 18218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
I20260925 04:38:00.722723 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.722506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46318)
0925 04:38:00.722610 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.722660 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:38:00.724121 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:38:00.730014 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.720338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:40008)
0925 04:38:00.720541 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.720558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.720657 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.720958 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.720962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.721130 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.721267 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.721277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.722407 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.722411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.723990 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.724006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.724930 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.724932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.725016 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.725659 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.725679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.726835 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.728680 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.728683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.728684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.728871 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.729138 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.729140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.729141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.729215 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.729611 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.729614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.729695 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.729778 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.729877 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 04:38:00.730266 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.720388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.27.46.253:40811)
0925 04:38:00.720504 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.720508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.720630 (+ 122us) server_negotiation.cc:407] Connection header received
0925 04:38:00.720736 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.720741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.720783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.720935 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.721404 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.722255 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.724360 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.724793 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.724909 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.726977 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.726981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.726983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.727014 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.728571 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.728966 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.728969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.728971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.729048 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.729300 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.729303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.729304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.729489 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.729611 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.729851 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.729973 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:38:00.732185 18194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:00.732877 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.732646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46332)
0925 04:38:00.732763 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.732812 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:38:00.733636 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.733346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46338)
0925 04:38:00.733485 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.733589 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260925 04:38:00.733865 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
I20260925 04:38:00.765044 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 1
I20260925 04:38:00.765959 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.765657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46340)
0925 04:38:00.765813 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.765881 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 04:38:00.766764 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.766567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46352)
0925 04:38:00.766670 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.766714 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 04:38:00.766957 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
I20260925 04:38:00.773418 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 1
I20260925 04:38:00.774161 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.773889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46364)
0925 04:38:00.774029 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.774093 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 04:38:00.774942 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.774723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46368)
0925 04:38:00.774834 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.774882 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:38:00.775120 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
W20260925 04:38:00.827406 18277 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:00.827668 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:00.827690 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:00.827724 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:00.831576 18277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:00.831616 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:00.831632 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:00.831663 18277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:00.831676 18277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:00.837241 18277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:34223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:35361,127.27.46.253:40811,127.27.46.252:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:00.838274 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:00.839389 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:38:00.842123 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 2
I20260925 04:38:00.842952 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.842678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46384)
0925 04:38:00.842820 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.842877 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 04:38:00.843312 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 2
I20260925 04:38:00.844054 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.843718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46400)
0925 04:38:00.843906 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.843976 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 04:38:00.844322 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.844046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46412)
0925 04:38:00.844197 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.844254 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 04:38:00.844491 18194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
I20260925 04:38:00.844826 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.844630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46422)
0925 04:38:00.844726 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.844776 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 04:38:00.844950 18271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:46315: Network error: Client connection negotiation failed: client connection to 127.27.46.252:46315: connect: Connection refused (error 111)
W20260925 04:38:00.848057 18287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.848054 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:00.848054 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:00.848328 18277 server_base.cc:1034] running on GCE node
I20260925 04:38:00.848867 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:00.850004 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:00.851214 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790311080851200 us; error 24 us; skew 500 ppm
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:00.853741 18277 init.cc:377] Logged in from keytab as kudu/127.27.46.252@KRBTEST.COM (short username kudu)
I20260925 04:38:00.854596 18277 webserver.cc:460] Webserver started at http://127.27.46.252:34237/ using document root <none> and password file <none>
I20260925 04:38:00.855089 18277 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:00.855177 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:00.855405 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:00.857278 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data/instance:
uuid: "02f895fcf0d54c1396e8c106a4eb217b"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "e65368d55acee721f205a5ea2006a01e"
server_key_iv: "00520f93453164e6b157110d57efa9e8"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.857813 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal/instance:
uuid: "02f895fcf0d54c1396e8c106a4eb217b"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "e65368d55acee721f205a5ea2006a01e"
server_key_iv: "00520f93453164e6b157110d57efa9e8"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.861040 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:38:00.863538 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:00.864756 18277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:00.864908 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "02f895fcf0d54c1396e8c106a4eb217b"
format_stamp: "Formatted at 2026-09-25 04:38:00 on dist-test-slave-r8r3"
server_key: "e65368d55acee721f205a5ea2006a01e"
server_key_iv: "00520f93453164e6b157110d57efa9e8"
server_key_version: "encryptionkey@0"
I20260925 04:38:00.865015 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:00.878424 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:00.881218 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:00.881444 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:00.888118 18277 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.252:46315
I20260925 04:38:00.888396 18346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.252:46315 every 8 connection(s)
I20260925 04:38:00.889014 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
I20260925 04:38:00.891721 18347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:00.893221 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18277
I20260925 04:38:00.893328 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790310801918616-27835-0/minicluster-data/master-2/wal/instance
I20260925 04:38:00.893625 27835 external_mini_cluster.cc:1428] Setting key cc7942ff70e4cd0bd82f8fc00a2c8a34
I20260925 04:38:00.895896 18347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:00.907280 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.897968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.27.46.254:35361)
0925 04:38:00.898095 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.898100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.898360 (+ 260us) server_negotiation.cc:407] Connection header received
0925 04:38:00.898516 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.898519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.898567 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.898653 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.899416 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.899936 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.901022 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.901248 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.901389 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.903885 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.903891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.903897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.903932 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.905581 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.906032 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.906035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.906036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.906131 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.906394 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.906398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.906399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.906574 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.906678 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.907008 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.907115 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 04:38:00.907956 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.894946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.27.46.252:46315)
0925 04:38:00.895329 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.895351 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.895385 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:38:00.896049 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.896067 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.896373 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.896704 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.897594 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.898825 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.899794 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.900354 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.900468 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.903264 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.903274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.903303 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.903330 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.905328 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.905810 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.905815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.905816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.905909 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.906187 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.906190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.906195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.906510 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.906717 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.906943 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.907214 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":138,"threads_started":1}
I20260925 04:38:00.907955 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.897871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:35361 (local address 127.0.0.1:50472)
0925 04:38:00.898243 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.898281 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.898418 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.898701 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.898707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.899133 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.899276 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.899284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.900080 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.900082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.900887 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.900898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.901454 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.901459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.901586 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.902078 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.902092 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.903725 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.905687 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.905696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.905704 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.905924 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.906222 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.906225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.906227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.906305 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.906678 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.906685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.906811 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.906956 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.907315 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":105,"thread_start_us":118,"threads_started":1}
I20260925 04:38:00.909180 18347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:00.917730 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.909767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:40811 (local address 127.0.0.1:40012)
0925 04:38:00.909859 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.909871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.909934 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.910229 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.910232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.910365 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.910495 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.910505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.911368 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.911370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.911918 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.911927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.912433 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.912440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.912549 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.913046 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.913059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.914367 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.916130 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.916134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.916137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.916361 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.916662 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.916665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.916667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.916737 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.917186 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.917191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.917288 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.917394 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.917551 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 04:38:00.917783 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.909863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.27.46.253:40811)
0925 04:38:00.909987 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.910009 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.910022 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:38:00.910063 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.910068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.910112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.910193 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.910637 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.911227 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.912028 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.912299 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.912383 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.914498 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.914500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.914502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.914525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.916013 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.916466 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.916472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.916476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.916563 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.916852 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.916856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.916858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.917083 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.917156 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.917473 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.917627 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:38:00.918557 18347 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:00.918650 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.910135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.27.46.254:35361)
0925 04:38:00.910247 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.910250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.910262 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:38:00.910297 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.910299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.910332 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.910399 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.911044 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.911607 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.912522 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.912717 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.912933 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.915368 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.915374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.915377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.915406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.916918 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.917361 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.917363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.917365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.917475 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.917785 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.917790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.917794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.918005 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.918239 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.918371 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.918496 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:38:00.927186 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.919061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46432)
0925 04:38:00.919251 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.919265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.919341 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.919564 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.919567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.919741 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.919813 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.919818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.920692 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.920696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.921522 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.921532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.922128 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.922131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.922227 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.922638 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.922649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.923984 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.925799 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.925802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.925804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.925965 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.926236 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.926238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.926242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.926309 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.926674 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.926677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.926758 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.926872 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.927012 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":17}
I20260925 04:38:00.927192 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.919147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.27.46.252:46315)
0925 04:38:00.919241 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.919244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.919304 (+ 60us) server_negotiation.cc:407] Connection header received
0925 04:38:00.919419 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.919421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.919456 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.919514 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.919922 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.920583 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.921674 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.922012 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.922120 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.924131 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.924135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.924140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.924174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.925689 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.926058 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.926064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.926066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.926134 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.926396 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.926401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.926402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.926568 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.926676 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.926933 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.927061 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":27}
I20260925 04:38:00.928716 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.919950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.27.46.253:40811)
0925 04:38:00.920106 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.920110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.920122 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:38:00.920155 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.920158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.920199 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.920274 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.920830 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.921603 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.922535 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.922813 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.922892 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.925577 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.925580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.925581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.925601 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.926994 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.927521 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.927524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.927525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.927597 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.927909 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.927913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.927915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.928077 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.928333 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.928451 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.928562 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:38:00.931025 18347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: Bootstrap starting.
I20260925 04:38:00.932049 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:38:00.933666 18347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:00.933286 18168 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "33f1c29198094109b49fbf8d122d9845"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50484
I20260925 04:38:00.934489 18347 log.cc:824] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:00.936117 18271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 3
I20260925 04:38:00.936347 18347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: No bootstrap required, opened a new log
I20260925 04:38:00.940022 18347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.940335 18347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f895fcf0d54c1396e8c106a4eb217b, State: Initialized, Role: FOLLOWER
I20260925 04:38:00.940692 18347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
I20260925 04:38:00.941237 18354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.941397 18354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:00.942034 18347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:00.943657 18194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } attempt: 3
I20260925 04:38:00.945417 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.936473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46442)
0925 04:38:00.936592 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.936607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.936684 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.936921 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.936924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.937073 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.937160 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.937166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.938091 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.938094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.938911 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.938922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.939500 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.939504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.939580 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.940110 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.940125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.941669 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.943673 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.943678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.943682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.943928 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.944315 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.944319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.944321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.944423 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.944919 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.944922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.945036 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.945120 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.945249 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 04:38:00.946235 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.936513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.27.46.252:46315)
0925 04:38:00.936629 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.936632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.936643 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:38:00.936772 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.936774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.936812 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.936880 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.937264 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.937960 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.939039 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.939375 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.939468 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.941810 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.941813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.941815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.941840 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.943519 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.944056 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.944059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.944060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.944199 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.944545 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.944560 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.944561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.944801 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.945912 (+ 1111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.945988 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.946093 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:38:00.949335 18365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:00.949414 18365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:00.949431 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: Bootstrap starting.
I20260925 04:38:00.945806 18277 master_runner.cc:186] Error getting master registration for 127.27.46.254:35361: OK, instance_id { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" instance_seqno: 1790311080478852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:38:00.949882 18277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:38:00.951494 18271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:00.952142 18271 log.cc:824] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:38:00 dist-test-slave-r8r3 krb5kdc[18109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311080, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
I20260925 04:38:00.954493 18271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: No bootstrap required, opened a new log
I20260925 04:38:00.957209 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.944074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:46315 (local address 127.0.0.1:46446)
0925 04:38:00.944195 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:00.944210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:00.944301 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:00.944859 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:00.944864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:00.945058 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:00.945159 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.945166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.946295 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.946298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.947115 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:00.947129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.947818 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.947822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:00.947925 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.948387 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:00.948401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:00.953299 (+ 4898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:00.955471 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.955474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.955476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.955752 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.956111 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:00.956114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:00.956116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:00.956226 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:00.956660 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:00.956664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:00.956768 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 04:38:00.956883 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 04:38:00.957030 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 04:38:00.957562 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.944305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.27.46.252:46315)
0925 04:38:00.944599 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.944608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.944621 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:38:00.944651 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.944653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.944707 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.945270 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.945396 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.946174 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.947311 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.947674 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.948954 (+ 1280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:00.953485 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:00.953488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:00.953513 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:00.953537 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:00.955333 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.955869 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.955873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.955874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.956002 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:00.956348 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:00.956351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:00.956353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:00.956560 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:00.956662 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:00.956975 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 04:38:00.957097 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260925 04:38:00.958666 18271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.958884 18271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85aa19e21ba646c882809d2009a42ecf, State: Initialized, Role: FOLLOWER
I20260925 04:38:00.959189 18271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.959668 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.959992 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:00.960528 18271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:00.960724 18194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: Bootstrap starting.
I20260925 04:38:00.964115 18194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:00.964983 18194 log.cc:824] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: Log is configured to *not* fsync() on all Append() calls
W20260925 04:38:00.965770 18381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:00.965886 18381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:00.966954 18194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: No bootstrap required, opened a new log
I20260925 04:38:00.963979 18201 master_runner.cc:186] Error getting master registration for 127.27.46.254:35361: OK, instance_id { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" instance_seqno: 1790311080478852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:38:00.967180 18201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:38:00.969693 18194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.969894 18194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f1c29198094109b49fbf8d122d9845, State: Initialized, Role: FOLLOWER
I20260925 04:38:00.970229 18194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.970752 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.971022 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:00.971443 18194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:00.971498 18168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:38:00.971604 18168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:00.972834 18168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.973779 18168 leader_election.cc:288] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [CANDIDATE]: Term 1 election: Requested vote from peers 85aa19e21ba646c882809d2009a42ecf (127.27.46.253:40811), 02f895fcf0d54c1396e8c106a4eb217b (127.27.46.252:46315)
I20260925 04:38:00.975378 18245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33f1c29198094109b49fbf8d122d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85aa19e21ba646c882809d2009a42ecf"
I20260925 04:38:00.975625 18245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [term 0 FOLLOWER]: Advancing to term 1
W20260925 04:38:00.976581 18395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:00.976640 18395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:00.976791 18245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f1c29198094109b49fbf8d122d9845 in term 1.
I20260925 04:38:00.977218 18142 leader_election.cc:302] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f1c29198094109b49fbf8d122d9845, 85aa19e21ba646c882809d2009a42ecf; no voters:
I20260925 04:38:00.977205 18321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33f1c29198094109b49fbf8d122d9845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "02f895fcf0d54c1396e8c106a4eb217b"
I20260925 04:38:00.977550 18384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 1 FOLLOWER]: Leader election won for term 1
W20260925 04:38:00.977643 18145 tablet.cc:2369] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:38:00.977758 18384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [term 1 LEADER]: Becoming Leader. State: Replica: 33f1c29198094109b49fbf8d122d9845, State: Running, Role: LEADER
I20260925 04:38:00.978044 18384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } }
I20260925 04:38:00.979773 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f1c29198094109b49fbf8d122d9845. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.979877 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:00.980226 18403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:38:00.977460 18321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:00.981128 18403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:00.981693 18321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f1c29198094109b49fbf8d122d9845 in term 1.
I20260925 04:38:00.985348 18122 master_runner.cc:263] No leader master found from master 33f1c29198094109b49fbf8d122d9845
I20260925 04:38:00.985406 18122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:38:00.987262 18321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [term 1 FOLLOWER]: Refusing update from remote peer 33f1c29198094109b49fbf8d122d9845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:00.987845 18384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:00.991807 18354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f1c29198094109b49fbf8d122d9845. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.991938 18354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:00.992746 18245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [term 1 FOLLOWER]: Refusing update from remote peer 33f1c29198094109b49fbf8d122d9845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:00.993307 18384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:00.997000 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33f1c29198094109b49fbf8d122d9845. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:00.997117 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:01.009243 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.986187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.27.46.254:35361)
0925 04:38:00.986286 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.986290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.993890 (+ 7600us) server_negotiation.cc:407] Connection header received
0925 04:38:00.994033 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.994035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.994075 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.994120 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.997953 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.998633 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.999668 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:01.000016 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:01.000087 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:01.002713 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:01.002717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:01.002720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:01.002746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:01.004420 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.007044 (+ 2624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.007047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.007049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.007131 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.007777 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.007780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.007782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.007965 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:01.008074 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:01.008983 (+ 909us) server_negotiation.cc:299] Negotiation successful
0925 04:38:01.009097 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 04:38:01.010035 18401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:01.010169 18401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:01.010470 18354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:01.010600 18354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:01.010793 18404 mvcc.cc:205] Tried to move back new op lower bound from 7333114187717337088 to 7333114187689533440. Current Snapshot: MvccSnapshot[applied={T|T < 7333114187717337088}]
I20260925 04:38:01.010921 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.981909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.27.46.253:40811)
0925 04:38:00.982018 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.982022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.992460 (+ 10438us) server_negotiation.cc:407] Connection header received
0925 04:38:00.992633 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.992635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.992692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.992738 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:00.994982 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.995670 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.996757 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:00.997075 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:00.997154 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:01.004846 (+ 7692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:01.004850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:01.004853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:01.004880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:01.006460 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.007417 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.007431 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.007433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.007506 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.008344 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.008348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.008349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.008538 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:01.008650 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:01.010634 (+ 1984us) server_negotiation.cc:299] Negotiation successful
0925 04:38:01.010775 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 04:38:01.011075 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:00.986590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.27.46.252:46315)
0925 04:38:00.986695 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:00.986698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:00.998140 (+ 11442us) server_negotiation.cc:407] Connection header received
0925 04:38:00.998264 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:00.998267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:00.998315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:00.998390 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.001600 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:01.002368 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:01.003511 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:01.003893 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:01.003968 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:01.005543 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:01.005549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:01.005552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:01.005596 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:01.007845 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.009591 (+ 1746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.009595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.009596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.009685 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:01.010052 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:01.010058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:01.010060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:01.010279 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:01.010405 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:01.010747 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 04:38:01.010889 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:38:01.011142 18408 mvcc.cc:205] Tried to move back new op lower bound from 7333114187717337088 to 7333114187689533440. Current Snapshot: MvccSnapshot[applied={T|T < 7333114187717337088}]
I20260925 04:38:01.014031 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:01.014132 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:01.015686 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33f1c29198094109b49fbf8d122d9845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f1c29198094109b49fbf8d122d9845" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 35361 } } peers { permanent_uuid: "85aa19e21ba646c882809d2009a42ecf" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 40811 } } peers { permanent_uuid: "02f895fcf0d54c1396e8c106a4eb217b" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 46315 } } }
I20260925 04:38:01.015790 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:01.016820 18410 mvcc.cc:205] Tried to move back new op lower bound from 7333114187717337088 to 7333114187689533440. Current Snapshot: MvccSnapshot[applied={T|T < 7333114187717337088}]
I20260925 04:38:01.022003 18403 catalog_manager.cc:1347] Generated new cluster ID: 4ade951cff534696afe39477cee82e1b
I20260925 04:38:01.022075 18403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:01.067466 18403 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:01.068737 18403 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:01.093382 18403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33f1c29198094109b49fbf8d122d9845: Generated new TSK 0
I20260925 04:38:01.094285 18403 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:38:01.126401 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.123063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.27.46.253:40811)
0925 04:38:01.123193 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.123198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.125222 (+ 2024us) server_negotiation.cc:407] Connection header received
0925 04:38:01.125434 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.125438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.125489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.125582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.126248 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40036: BlockingRecv error: recv got EOF from 127.0.0.1:40036 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 04:38:01.128400 18198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.123921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.27.46.254:35361)
0925 04:38:01.124043 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.124049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.126894 (+ 2845us) server_negotiation.cc:407] Connection header received
0925 04:38:01.127422 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.127426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.127478 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.127533 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.128281 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50506: BlockingRecv error: recv got EOF from 127.0.0.1:50506 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 04:38:01.129163 18366 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.124699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.27.46.252:46315)
0925 04:38:01.124837 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.124841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.127099 (+ 2258us) server_negotiation.cc:407] Connection header received
0925 04:38:01.127686 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.127689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.127741 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.127793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.129049 (+ 1256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46458: BlockingRecv error: recv got EOF from 127.0.0.1:46458 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 04:38:01.133349 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.131813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.27.46.253:40811)
0925 04:38:01.131912 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.131916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.132378 (+ 462us) server_negotiation.cc:407] Connection header received
0925 04:38:01.132607 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.132611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.132703 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.132760 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.133251 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40050: BlockingRecv error: recv got EOF from 127.0.0.1:40050 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 04:38:01.138401 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.136830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.27.46.253:40811)
0925 04:38:01.136927 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.136930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.137441 (+ 511us) server_negotiation.cc:407] Connection header received
0925 04:38:01.137694 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.137697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.137748 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.137801 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.138297 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40066: BlockingRecv error: recv got EOF from 127.0.0.1:40066 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 04:38:01.143790 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.142268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.27.46.253:40811)
0925 04:38:01.142358 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.142362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.142861 (+ 499us) server_negotiation.cc:407] Connection header received
0925 04:38:01.143054 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.143057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.143103 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.143152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.143697 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40082: BlockingRecv error: recv got EOF from 127.0.0.1:40082 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 04:38:01.154558 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.149600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.27.46.253:40811)
0925 04:38:01.149738 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.149742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.153791 (+ 4049us) server_negotiation.cc:407] Connection header received
0925 04:38:01.153968 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.153971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.154009 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.154048 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.154479 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40090: BlockingRecv error: recv got EOF from 127.0.0.1:40090 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 04:38:01.166021 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.164654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.27.46.253:40811)
0925 04:38:01.164753 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.164758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.165190 (+ 432us) server_negotiation.cc:407] Connection header received
0925 04:38:01.165355 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.165358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.165400 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.165442 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.165928 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40104: BlockingRecv error: recv got EOF from 127.0.0.1:40104 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 04:38:01.185390 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.184182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.27.46.253:40811)
0925 04:38:01.184321 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.184328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.184550 (+ 222us) server_negotiation.cc:407] Connection header received
0925 04:38:01.184705 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.184709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.184764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.184847 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.185254 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40114: BlockingRecv error: recv got EOF from 127.0.0.1:40114 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 04:38:01.220602 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.219584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.27.46.253:40811)
0925 04:38:01.219753 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.219758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.219796 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:38:01.219934 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.219936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.219987 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.220091 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.220504 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40126: BlockingRecv error: recv got EOF from 127.0.0.1:40126 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 04:38:01.288235 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.287080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.27.46.253:40811)
0925 04:38:01.287263 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.287273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.287374 (+ 101us) server_negotiation.cc:407] Connection header received
0925 04:38:01.287496 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.287498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.287547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.287616 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.288086 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40130: BlockingRecv error: recv got EOF from 127.0.0.1:40130 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 04:38:01.419600 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.418594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.27.46.253:40811)
0925 04:38:01.418756 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.418760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.418854 (+ 94us) server_negotiation.cc:407] Connection header received
0925 04:38:01.418988 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.418990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.419035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.419111 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.419504 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40138: BlockingRecv error: recv got EOF from 127.0.0.1:40138 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 04:38:01.679018 18276 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:01.677809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.27.46.253:40811)
0925 04:38:01.677964 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:01.677968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:01.678073 (+ 105us) server_negotiation.cc:407] Connection header received
0925 04:38:01.678272 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:01.678275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:01.678338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:01.678418 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:01.678898 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40150: BlockingRecv error: recv got EOF from 127.0.0.1:40150 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:38:01.950850 18365 catalog_manager.cc:1259] Loaded cluster ID: 4ade951cff534696afe39477cee82e1b
I20260925 04:38:01.950934 18365 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: loading cluster ID for follower catalog manager: success
I20260925 04:38:01.952430 18365 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: acquiring CA information for follower catalog manager: success
I20260925 04:38:01.953657 18365 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02f895fcf0d54c1396e8c106a4eb217b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:38:01.967118 18381 catalog_manager.cc:1259] Loaded cluster ID: 4ade951cff534696afe39477cee82e1b
I20260925 04:38:01.967183 18381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: loading cluster ID for follower catalog manager: success
I20260925 04:38:01.969064 18381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: acquiring CA information for follower catalog manager: success
I20260925 04:38:01.970256 18381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85aa19e21ba646c882809d2009a42ecf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:38:02.196322 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:02.193024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.27.46.253:40811)
0925 04:38:02.193252 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:02.193256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:02.193273 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:38:02.193424 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:02.193426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:02.193561 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:02.193658 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:02.193914 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.194356 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.195085 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.195273 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.195342 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:02.195618 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:02.196027 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:02.196113 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:02.196167 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:38:02.196210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260925 04:38:02.202397 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:02.199217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.27.46.252:46315)
0925 04:38:02.199428 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:02.199432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:02.199444 (+ 12us) server_negotiation.cc:407] Connection header received
0925 04:38:02.199513 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:02.199515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:02.199607 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:02.199692 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:02.199874 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.200386 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.201130 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.201373 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.201457 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:02.201756 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:02.202101 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:02.202183 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:02.202237 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:38:02.202290 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260925 04:38:02.208773 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:02.205138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.27.46.254:35361)
0925 04:38:02.205417 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:02.205424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:02.205437 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:38:02.205553 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:02.205558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:02.205703 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:02.205826 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:02.206015 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.206668 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.207395 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.207637 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.207719 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:02.208019 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:02.208286 (+ 267us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:02.208377 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:02.208466 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:38:02.208520 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260925 04:38:02.208925 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:02.205164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.27.46.253:40811)
0925 04:38:02.205275 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:02.205280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:02.205400 (+ 120us) server_negotiation.cc:407] Connection header received
0925 04:38:02.205544 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:02.205547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:02.205618 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:02.205680 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:02.205936 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.206606 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.207623 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.207866 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.207977 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:02.208243 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:02.208317 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:02.208373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:02.208729 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:38:02.208777 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:38:02.209178 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:02.205544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.27.46.252:46315)
0925 04:38:02.205670 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:02.205674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:02.206109 (+ 435us) server_negotiation.cc:407] Connection header received
0925 04:38:02.206204 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:02.206207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:02.206253 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:02.206332 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:02.206533 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.207131 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.208131 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:02.208397 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:02.208480 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:02.208777 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:02.208845 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:02.208911 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:02.209041 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 04:38:02.209081 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:38:02.217440 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18122
I20260925 04:38:02.224632 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18201
I20260925 04:38:02.230460 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18277
2026-09-25T04:38:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3030 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:38:02 dist-test-slave-r8r3 krb5kdc[18499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:38:02 dist-test-slave-r8r3 krb5kdc[18499](info): set up 2 sockets
Sep 25 04:38:02 dist-test-slave-r8r3 krb5kdc[18499](info): commencing operation
krb5kdc: starting...
W20260925 04:38:03.229238 27835 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/test-user.keytab.
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
Entry for principal kudu/127.27.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab.
I20260925 04:38:03.348855 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38005
--webserver_interface=127.27.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:38:03.450641 18512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:03.450902 18512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:03.450924 18512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:03.451040 18512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:03.454559 18512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:03.454602 18512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:03.454617 18512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:03.454632 18512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:03.454667 18512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:03.459002 18512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.254.keytab
--principal=kudu/127.27.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.254:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:03.459890 18512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:03.461057 18512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:03.468724 18517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:03.468750 18520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:03.468807 18512 server_base.cc:1034] running on GCE node
W20260925 04:38:03.468750 18518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:03.469409 18512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:03.470489 18512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:03.471660 18512 hybrid_clock.cc:648] HybridClock initialized: now 1790311083471627 us; error 45 us; skew 500 ppm
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:03.474195 18512 init.cc:377] Logged in from keytab as kudu/127.27.46.254@KRBTEST.COM (short username kudu)
I20260925 04:38:03.474996 18512 webserver.cc:460] Webserver started at http://127.27.46.254:39433/ using document root <none> and password file <none>
I20260925 04:38:03.475489 18512 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:03.475543 18512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:03.475695 18512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:03.477417 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data/instance:
uuid: "d31a02735bfd463c934dcef271464064"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "59eddb373fb44e9253b4773db12127e2"
server_key_iv: "3336714705a67e7d6afbec3fa6879db2"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.478013 18512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance:
uuid: "d31a02735bfd463c934dcef271464064"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "59eddb373fb44e9253b4773db12127e2"
server_key_iv: "3336714705a67e7d6afbec3fa6879db2"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.481336 18512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:38:03.483644 18527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:03.484946 18512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 04:38:03.485114 18512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal
uuid: "d31a02735bfd463c934dcef271464064"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "59eddb373fb44e9253b4773db12127e2"
server_key_iv: "3336714705a67e7d6afbec3fa6879db2"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.485213 18512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:03.508047 18512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:03.511372 18512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:03.511584 18512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:03.518363 18512 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.254:38005
I20260925 04:38:03.518625 18581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.254:38005 every 8 connection(s)
I20260925 04:38:03.519304 18512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/data/info.pb
I20260925 04:38:03.522032 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:03.524893 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18512
I20260925 04:38:03.525027 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-0/wal/instance
I20260925 04:38:03.525264 27835 external_mini_cluster.cc:1428] Setting key 73c7f11d159e64b8799e5d179b0b0dc8
I20260925 04:38:03.525745 18582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.253@KRBTEST.COM" created.
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:03.543094 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.527964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38005 (local address 127.0.0.1:47300)
0925 04:38:03.528613 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.528663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.529921 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.530660 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.530668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.531185 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.531605 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.531627 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.532743 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.532746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.533962 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.533972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.534633 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.534635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.534730 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.535191 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.535206 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.538042 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.540168 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.540176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.540182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.540424 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.540851 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.540854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.540856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.540995 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.541795 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.541799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.541911 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.542031 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.542270 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"spinlock_wait_cycles":124672,"thread_start_us":214,"threads_started":1}
I20260925 04:38:03.543098 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.527961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.27.46.254:38005)
0925 04:38:03.528440 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.528458 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.528798 (+ 340us) server_negotiation.cc:407] Connection header received
0925 04:38:03.530257 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.530261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.530444 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.530586 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.531782 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.532602 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.534077 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.534481 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.534593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.538214 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.538219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.538228 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.538248 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.540062 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.540557 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.540567 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.540571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.540720 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.541110 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.541116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.541121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.541675 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.541817 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.542178 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.542358 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":183,"threads_started":1}
I20260925 04:38:03.545598 18582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:03.546350 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.546085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55142)
0925 04:38:03.546228 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.546294 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 04:38:03.546543 18529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
I20260925 04:38:03.547430 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.547210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55152)
0925 04:38:03.547345 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.547385 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 04:38:03.547580 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:43805: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
Entry for principal kudu/127.27.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab.
I20260925 04:38:03.556977 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:43805
--webserver_interface=127.27.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:38:03.596848 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } attempt: 1
I20260925 04:38:03.597955 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.597606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55166)
0925 04:38:03.597782 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.597857 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:38:03.598719 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.598494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55182)
0925 04:38:03.598602 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.598652 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 04:38:03.598873 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:43805: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
W20260925 04:38:03.661652 18589 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:03.661968 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:03.662016 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:03.662106 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:38:03.666079 18589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:03.666126 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:03.666144 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:03.666186 18589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:03.666220 18589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:03.667069 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } attempt: 2
I20260925 04:38:03.667917 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.667602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55196)
0925 04:38:03.667787 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.667846 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 04:38:03.668581 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.668405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55198)
0925 04:38:03.668495 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.668535 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 04:38:03.668718 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.253:43805: Network error: Client connection negotiation failed: client connection to 127.27.46.253:43805: connect: Connection refused (error 111)
I20260925 04:38:03.671742 18589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.253.keytab
--principal=kudu/127.27.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.253:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:03.672673 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:03.673655 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 04:38:03.682034 18589 server_base.cc:1034] running on GCE node
W20260925 04:38:03.682016 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:03.681974 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:03.681982 18597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:03.682807 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:03.683888 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:03.685084 18589 hybrid_clock.cc:648] HybridClock initialized: now 1790311083685072 us; error 23 us; skew 500 ppm
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:03.687700 18589 init.cc:377] Logged in from keytab as kudu/127.27.46.253@KRBTEST.COM (short username kudu)
I20260925 04:38:03.688592 18589 webserver.cc:460] Webserver started at http://127.27.46.253:37533/ using document root <none> and password file <none>
I20260925 04:38:03.689167 18589 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:03.689260 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:03.689487 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:03.691570 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data/instance:
uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "d137d40769f2f888627d71573a020c4a"
server_key_iv: "cde755809463ad909e91057c02864462"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.692134 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal/instance:
uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "d137d40769f2f888627d71573a020c4a"
server_key_iv: "cde755809463ad909e91057c02864462"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.695706 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 04:38:03.698103 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:03.699154 18589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:38:03.699301 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal
uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "d137d40769f2f888627d71573a020c4a"
server_key_iv: "cde755809463ad909e91057c02864462"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.699409 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:03.720645 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:03.723387 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:03.723596 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:03.730229 18589 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.253:43805
I20260925 04:38:03.730504 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.253:43805 every 8 connection(s)
I20260925 04:38:03.731215 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/data/info.pb
I20260925 04:38:03.732796 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18589
I20260925 04:38:03.732944 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-1/wal/instance
I20260925 04:38:03.733222 27835 external_mini_cluster.cc:1428] Setting key fb1dfe2d43d8d2a248575b7d10282660
I20260925 04:38:03.734280 18662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:03.738145 18662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.46.252@KRBTEST.COM" created.
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:03.752127 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.739957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.27.46.254:38005)
0925 04:38:03.740134 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.740141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.740903 (+ 762us) server_negotiation.cc:407] Connection header received
0925 04:38:03.741924 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.741929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.741995 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.742103 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.743199 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.743764 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.745042 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.745268 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.745349 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.748139 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.748147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.748149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.748172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.749842 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.750372 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.750375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.750376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.750452 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.750779 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.750785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.750787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.751070 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.751198 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.751795 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.751936 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:38:03.752761 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.740286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38005 (local address 127.0.0.1:47302)
0925 04:38:03.740735 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.740785 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.741708 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.742244 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.742250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.742569 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.743057 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.743070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.743952 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.743959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.744880 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.744908 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.745403 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.745406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.745691 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.746304 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.746320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.747969 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.749965 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.749969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.749980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.750252 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.750543 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.750545 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.750546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.750674 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.751190 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.751195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.751433 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.751698 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.751926 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260925 04:38:03.754201 18662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:38:03.759927 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } attempt: 3
I20260925 04:38:03.763484 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.754692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55208)
0925 04:38:03.754871 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.754904 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.754975 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.755401 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.755403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.755604 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.755680 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.755684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.756646 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.756652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.757152 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.757160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.757838 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.757841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.757945 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.758424 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.758437 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.759560 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.761807 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.761811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.761813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.762075 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.762344 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.762346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.762347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.762441 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.762874 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.762878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.762983 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.763098 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.763266 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 04:38:03.763505 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.754803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.27.46.253:43805)
0925 04:38:03.755109 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.755117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.755133 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:38:03.755160 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.755164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.755293 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.755421 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.755789 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.756531 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.757278 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.757718 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.757831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.759677 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.759686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.759697 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.759716 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.761720 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.762175 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.762181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.762182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.762264 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.762547 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.762550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.762551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.762763 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.762899 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.763156 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.763353 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":215,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
Entry for principal kudu/127.27.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab.
I20260925 04:38:03.764947 27835 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
/tmp/dist-test-taskjz8mnA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:38489
--webserver_interface=127.27.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.46.212:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:38:03.765941 18662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:38:03.766652 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.766388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59158)
0925 04:38:03.766518 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.766588 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 04:38:03.766872 18607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
I20260925 04:38:03.767760 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.767557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59170)
0925 04:38:03.767651 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.767696 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 04:38:03.767889 18662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
I20260925 04:38:03.769874 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.760447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.27.46.253:43805)
0925 04:38:03.760989 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.761000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.761015 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:38:03.761061 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.761063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.761105 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.761207 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.761676 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.762373 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.763981 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.764319 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.764424 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.766521 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.766527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.766529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.766549 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.768214 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.768721 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.768723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.768724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.768810 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.769097 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.769099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.769100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.769234 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.769301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.769654 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.769776 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":101,"threads_started":1}
I20260925 04:38:03.769960 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.760356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55224)
0925 04:38:03.760487 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.760501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.760620 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.761253 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.761257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.761411 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.761558 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.761565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.762483 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.762488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.763810 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.763820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.764457 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.764460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.764596 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.764996 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.765007 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.766387 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.768317 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.768322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.768326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.768634 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.768901 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.768906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.768911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.769008 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.769332 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.769338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.769450 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.769590 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.769734 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 04:38:03.770751 18582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:03.771502 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.771172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59184)
0925 04:38:03.771324 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.771406 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:38:03.772274 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.771961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59198)
0925 04:38:03.772133 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.772192 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 04:38:03.772408 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
I20260925 04:38:03.803583 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 1
I20260925 04:38:03.804483 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.804137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59212)
0925 04:38:03.804316 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.804389 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 04:38:03.805291 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.805066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59220)
0925 04:38:03.805184 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.805232 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:38:03.805452 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
I20260925 04:38:03.817135 18662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 1
I20260925 04:38:03.818028 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.817745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59236)
0925 04:38:03.817887 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.817955 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:38:03.818785 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.818571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59238)
0925 04:38:03.818682 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.818732 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:38:03.818907 18662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
W20260925 04:38:03.878903 18669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 04:38:03.879242 18669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:38:03.879317 18669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:38:03.879420 18669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:38:03.880686 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 2
I20260925 04:38:03.881676 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.881263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59244)
0925 04:38:03.881469 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.881576 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 04:38:03.882460 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.882217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59254)
0925 04:38:03.882335 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.882393 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:38:03.882606 18582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
W20260925 04:38:03.884124 18669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:38:03.884183 18669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:38:03.884234 18669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:38:03.884279 18669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:38:03.884315 18669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:38:03.887054 18662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 2
I20260925 04:38:03.887773 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.887482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59270)
0925 04:38:03.887608 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.887666 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:38:03.888588 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.888362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59286)
0925 04:38:03.888477 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.888523 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:38:03.888737 18662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.46.252:38489: Network error: Client connection negotiation failed: client connection to 127.27.46.252:38489: connect: Connection refused (error 111)
I20260925 04:38:03.890228 18669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.46.212:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.46.254:38005,127.27.46.253:43805,127.27.46.252:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/krb5kdc/kudu_127.27.46.252.keytab
--principal=kudu/127.27.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.46.252:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 04:04:05 UTC on ccycloud.quasar-hscabb.root.comops.site
build id 4671825
I20260925 04:38:03.891359 18669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:38:03.892397 18669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:38:03.900935 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:03.900935 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:38:03.900988 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:38:03.901280 18669 server_base.cc:1034] running on GCE node
I20260925 04:38:03.901974 18669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:38:03.903137 18669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:38:03.904311 18669 hybrid_clock.cc:648] HybridClock initialized: now 1790311083904290 us; error 30 us; skew 500 ppm
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:38:03.907330 18669 init.cc:377] Logged in from keytab as kudu/127.27.46.252@KRBTEST.COM (short username kudu)
I20260925 04:38:03.908233 18669 webserver.cc:460] Webserver started at http://127.27.46.252:38143/ using document root <none> and password file <none>
I20260925 04:38:03.908756 18669 fs_manager.cc:359] Metadata directory not provided
I20260925 04:38:03.908820 18669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:38:03.908990 18669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:38:03.911085 18669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data/instance:
uuid: "3e0a6e0d484a4eddb53486119b29dcf0"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "f7e3f8046643823c2b2117b692cf87a5"
server_key_iv: "617cbe7350257535d3123b80a85507e7"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.911633 18669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal/instance:
uuid: "3e0a6e0d484a4eddb53486119b29dcf0"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "f7e3f8046643823c2b2117b692cf87a5"
server_key_iv: "617cbe7350257535d3123b80a85507e7"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.915444 18669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:38:03.917900 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:38:03.919289 18669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:38:03.919456 18669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data,/tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal
uuid: "3e0a6e0d484a4eddb53486119b29dcf0"
format_stamp: "Formatted at 2026-09-25 04:38:03 on dist-test-slave-r8r3"
server_key: "f7e3f8046643823c2b2117b692cf87a5"
server_key_iv: "617cbe7350257535d3123b80a85507e7"
server_key_version: "encryptionkey@0"
I20260925 04:38:03.919577 18669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:38:03.939795 18669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:38:03.943032 18669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:38:03.943238 18669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:38:03.949694 18669 rpc_server.cc:307] RPC server started. Bound to: 127.27.46.252:38489
I20260925 04:38:03.949997 18739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.46.252:38489 every 8 connection(s)
I20260925 04:38:03.950624 18669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/data/info.pb
I20260925 04:38:03.952461 27835 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu as pid 18669
I20260925 04:38:03.952589 27835 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjz8mnA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790310801918616-27835-0/minicluster-data/master-2/wal/instance
I20260925 04:38:03.952869 27835 external_mini_cluster.cc:1428] Setting key ddc9d22e4c69a816010b3d9cb8e5ad8f
I20260925 04:38:03.953598 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:38:03.957567 18740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
I20260925 04:38:03.966514 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.954153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.27.46.252:38489)
0925 04:38:03.954507 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.954530 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.954563 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:38:03.955164 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.955178 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.955485 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.955736 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.956611 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.957476 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.958550 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.959027 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.959204 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.961851 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.961856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.961864 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.961886 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.963695 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.964101 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.964109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.964115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.964264 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.964548 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.964551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.964553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.964978 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.965158 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.965493 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.965752 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20260925 04:38:03.967829 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.959246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.254:38005 (local address 127.0.0.1:47318)
0925 04:38:03.959605 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.959656 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.959775 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.960127 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.960133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.960452 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.960573 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.960579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.961368 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.961372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.962043 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.962051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.962583 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.962587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.962697 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.963047 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.963057 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.964544 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.966243 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.966249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.966254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.966491 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.966857 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.966861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.966862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.966949 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.967294 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.967299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.967416 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.967547 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.967706 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260925 04:38:03.967875 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.959335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.27.46.254:38005)
0925 04:38:03.959500 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.959507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.959747 (+ 240us) server_negotiation.cc:407] Connection header received
0925 04:38:03.959905 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.959908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.959968 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.960063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.960706 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.961225 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.962162 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.962478 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.962579 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.964688 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.964691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.964693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.964713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.966136 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.966615 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.966619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.966622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.966752 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.967034 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.967036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.967036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.967189 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.967269 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.967632 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.967744 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:38:03.968931 18740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.254@KRBTEST.COM
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
I20260925 04:38:03.976934 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.968883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.27.46.254:38005)
0925 04:38:03.968986 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.968990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.969001 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:38:03.969032 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.969034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.969065 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.969119 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.969793 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.970271 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.971036 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.971262 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.971376 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.973784 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.973787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.973789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.973808 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.975295 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.975707 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.975710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.975712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.975792 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.976092 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.976098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.976100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.976308 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.976452 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.976655 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.976766 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 04:38:03.977054 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.969406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.253:43805 (local address 127.0.0.1:55240)
0925 04:38:03.969572 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.969584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.969650 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.969935 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.969938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.970096 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.970173 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.970182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.970962 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.970968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.971473 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.971480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.971979 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.971982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.972065 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.972465 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.972474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.973796 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.975494 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.975497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.975498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.975711 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.976023 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.976026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.976027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.976143 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.976562 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.976587 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.976671 (+ 84us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.976783 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.976902 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 04:38:03.977288 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.969577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.27.46.253:43805)
0925 04:38:03.969737 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.969741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.969754 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:38:03.969799 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.969801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.969836 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.969924 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.970331 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.970851 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.971596 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.971845 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.971977 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.973945 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.973948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.973950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.973970 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.975389 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.975804 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.975806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.975807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.975924 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.976244 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.976246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.976247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.976451 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.976547 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.976963 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.977046 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:38:03.977970 18740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:38:03.979924 18662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 3
I20260925 04:38:03.981792 18582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } attempt: 3
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.252@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.46.253@KRBTEST.COM
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.253@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
I20260925 04:38:03.987721 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.978485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59312)
0925 04:38:03.978627 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.978641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.978720 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.978964 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.978967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.979122 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.979192 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.979197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.980046 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.980051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.980785 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.980795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.981359 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.981362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.981438 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.982051 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.982070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.983718 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.985995 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.985998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.986000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.986247 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.986524 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.986528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.986531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.986638 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.987094 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.987101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.987209 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.987323 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.987522 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"spinlock_wait_cycles":36224}
I20260925 04:38:03.988204 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.978542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.27.46.252:38489)
0925 04:38:03.978643 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.978650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.978684 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:38:03.978815 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.978818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.978864 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.978931 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.979315 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.979947 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.980906 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.981240 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.981332 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.983851 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.983854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.983856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.983883 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.985886 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.986353 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.986358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.986360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.986431 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.986739 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.986745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.986749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.986958 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.987058 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.987967 (+ 909us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.988072 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 04:38:03.988749 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.978623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.27.46.253:43805)
0925 04:38:03.978720 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.978724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.978770 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:38:03.978897 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.978899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.978961 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.979035 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.979703 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.980249 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.981229 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.981619 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.981723 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.984838 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.984841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.984843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.984866 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.986634 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.987098 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.987101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.987103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.987186 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.987483 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.987486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.987488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.987690 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.987881 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.988457 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.988571 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 04:38:03 dist-test-slave-r8r3 krb5kdc[18499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790311083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.46.254@KRBTEST.COM for kudu/127.27.46.252@KRBTEST.COM
I20260925 04:38:03.991906 27835 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:38:03.991958 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.980206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59324)
0925 04:38:03.980377 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.980394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.980443 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.981124 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.981127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.981311 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.981420 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.981426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.982979 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.982984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.983593 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.983602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.984297 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.984302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.984391 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.984814 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.984829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.986450 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.990276 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.990279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.990282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.990543 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.990884 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.990887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.990889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.990980 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.991496 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.991499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.991589 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.991669 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.991792 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 04:38:03.992158 18740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: Bootstrap starting.
I20260925 04:38:03.992553 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.980626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.27.46.252:38489)
0925 04:38:03.980908 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.980916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.980927 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:38:03.980954 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.980956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.980992 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.981073 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.981577 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.982891 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.983721 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.984162 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.984617 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.986581 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.986584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.986586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.986607 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.990129 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.990671 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.990675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.990676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.990776 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.991104 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.991110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.991115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.991367 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.992195 (+ 828us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.992275 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.992374 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"spinlock_wait_cycles":2432,"thread_start_us":106,"threads_started":1}
I20260925 04:38:03.993287 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.982160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.46.252:38489 (local address 127.0.0.1:59336)
0925 04:38:03.982488 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 04:38:03.982501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 04:38:03.982575 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:38:03.984234 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:38:03.984240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:38:03.984417 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:38:03.984530 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.984539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.986170 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.986174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.986998 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:38:03.987009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.987644 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.987647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:38:03.987731 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.988097 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:38:03.988113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:38:03.989491 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:38:03.991907 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.991911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.991913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.992131 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.992421 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:38:03.992424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:38:03.992426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:38:03.992496 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:38:03.992862 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:38:03.992865 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:38:03.992951 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 04:38:03.993024 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:38:03.993123 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260925 04:38:03.993834 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:03.982698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.27.46.252:38489)
0925 04:38:03.983948 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:03.983957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:03.983968 (+ 11us) server_negotiation.cc:407] Connection header received
0925 04:38:03.983998 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:03.984000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:03.984048 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:03.984199 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:03.985147 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.986031 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.987139 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:03.987529 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:03.987915 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:03.989705 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:03.989711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:03.989713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:03.989735 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:03.991804 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.992243 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.992245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.992246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.992310 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:03.992594 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:03.992598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:03.992602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:03.992767 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:03.993526 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:03.993596 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:38:03.993690 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":93,"threads_started":1}
I20260925 04:38:03.993005 18555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d31a02735bfd463c934dcef271464064"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47320
I20260925 04:38:03.995234 18740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:03.995311 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: Bootstrap starting.
I20260925 04:38:03.996071 18740 log.cc:824] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:03.996198 18582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: Bootstrap starting.
I20260925 04:38:03.997637 18662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:03.998143 18740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: No bootstrap required, opened a new log
I20260925 04:38:03.998359 18582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: Neither blocks nor log segments found. Creating new log.
I20260925 04:38:03.998389 18662 log.cc:824] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:03.999051 18582 log.cc:824] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: Log is configured to *not* fsync() on all Append() calls
I20260925 04:38:04.001003 18582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: No bootstrap required, opened a new log
I20260925 04:38:04.001936 18740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.002183 18740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0a6e0d484a4eddb53486119b29dcf0, State: Initialized, Role: FOLLOWER
I20260925 04:38:04.002519 18740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.003027 18750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.003160 18750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.003794 18740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:04.003954 18582 raft_consensus.cc:348] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.004159 18582 raft_consensus.cc:729] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31a02735bfd463c934dcef271464064, State: Initialized, Role: FOLLOWER
I20260925 04:38:04.004482 18582 consensus_queue.cc:260] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.004979 18755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.005208 18755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.005769 18582 sys_catalog.cc:564] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:04.005838 18555 raft_consensus.cc:482] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 04:38:04.005924 18555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:04.007120 18555 raft_consensus.cc:504] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.007493 18662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: No bootstrap required, opened a new log
I20260925 04:38:04.007949 18555 leader_election.cc:288] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [CANDIDATE]: Term 1 election: Requested vote from peers 8bc45c5cd0b8498ab8d6b55608ca2dcc (127.27.46.253:43805), 3e0a6e0d484a4eddb53486119b29dcf0 (127.27.46.252:38489)
W20260925 04:38:04.008657 18762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:04.008733 18762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:04.007421 18669 master_runner.cc:186] Error getting master registration for 127.27.46.254:38005: OK, instance_id { permanent_uuid: "d31a02735bfd463c934dcef271464064" instance_seqno: 1790311083516524 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:38:04.010154 18669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:38:04.010823 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31a02735bfd463c934dcef271464064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e0a6e0d484a4eddb53486119b29dcf0"
I20260925 04:38:04.011055 18713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:04.011055 18662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.011264 18662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc45c5cd0b8498ab8d6b55608ca2dcc, State: Initialized, Role: FOLLOWER
I20260925 04:38:04.011595 18662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.012423 18713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31a02735bfd463c934dcef271464064 in term 1.
I20260925 04:38:04.012874 18528 leader_election.cc:302] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e0a6e0d484a4eddb53486119b29dcf0, d31a02735bfd463c934dcef271464064; no voters:
W20260925 04:38:04.013229 18533 tablet.cc:2369] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:38:04.013794 18755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:38:04.014353 18755 raft_consensus.cc:686] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [term 1 LEADER]: Becoming Leader. State: Replica: d31a02735bfd463c934dcef271464064, State: Running, Role: LEADER
I20260925 04:38:04.014714 18755 consensus_queue.cc:237] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } }
I20260925 04:38:04.016314 18755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31a02735bfd463c934dcef271464064. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.016467 18755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:04.014721 18512 master_runner.cc:186] Error getting master registration for 127.27.46.253:43805: OK, instance_id { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" instance_seqno: 1790311083728388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:38:04.017581 18512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:38:04.017692 18774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.017824 18774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.018167 18662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:38:04.018556 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31a02735bfd463c934dcef271464064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc"
I20260925 04:38:04.018781 18635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:38:04.020175 18635 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31a02735bfd463c934dcef271464064 in term 1.
I20260925 04:38:04.022889 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:04.011702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.27.46.254:38005)
0925 04:38:04.011811 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.011815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.013040 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 04:38:04.013325 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.013328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.013373 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.013419 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.014863 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.015475 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.016818 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.017089 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.017172 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:04.019059 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:04.019062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:04.019065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:04.019088 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:04.020737 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.021279 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.021283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.021285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.021369 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.021711 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.021715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.021717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.021912 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:04.022023 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:04.022613 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 04:38:04.022733 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 04:38:04.027385 18798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:04.027465 18798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:04.029271 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:04.012417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.27.46.253:43805)
0925 04:38:04.012559 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.012564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.017691 (+ 5127us) server_negotiation.cc:407] Connection header received
0925 04:38:04.018054 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.018057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.018109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.018161 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.021286 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.022026 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.023435 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.023765 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.023841 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:04.025643 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:04.025646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:04.025649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:04.025673 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:04.027324 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.027943 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.027949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.027950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.028041 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.028355 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.028358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.028359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.028552 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:04.028650 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:04.029001 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 04:38:04.029121 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 04:38:04.030670 18781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:38:04.030740 18781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:38:04.031184 18785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:38:04.032018 18785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:38:04.035531 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:04.017031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.27.46.252:38489)
0925 04:38:04.017126 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.017132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.023665 (+ 6533us) server_negotiation.cc:407] Connection header received
0925 04:38:04.023836 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.023839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.023892 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.023945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.024868 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.025708 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.026877 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:04.027291 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:04.027374 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:04.032005 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:38:04.032008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:38:04.032011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:38:04.032035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:38:04.033729 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.034216 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.034220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.034222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.034313 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:38:04.034629 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:38:04.034632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:38:04.034640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:38:04.034857 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:38:04.035019 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:04.035262 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 04:38:04.035380 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 04:38:04.040910 18713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [term 1 FOLLOWER]: Refusing update from remote peer d31a02735bfd463c934dcef271464064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:04.042229 18635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [term 1 FOLLOWER]: Refusing update from remote peer d31a02735bfd463c934dcef271464064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:38:04.042589 18755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:04.043011 18755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:38:04.046447 18750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31a02735bfd463c934dcef271464064. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.046561 18750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.046901 18774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31a02735bfd463c934dcef271464064. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.047008 18774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.049820 18755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.049917 18755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:04.050143 18750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.050253 18750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.050473 18801 mvcc.cc:205] Tried to move back new op lower bound from 7333114200224374784 to 7333114200127406080. Current Snapshot: MvccSnapshot[applied={T|T < 7333114200224374784}]
I20260925 04:38:04.053355 18803 mvcc.cc:205] Tried to move back new op lower bound from 7333114200224374784 to 7333114200127406080. Current Snapshot: MvccSnapshot[applied={T|T < 7333114200224374784}]
I20260925 04:38:04.056346 18783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.056458 18783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064 [sys.catalog]: This master's current role is: LEADER
I20260925 04:38:04.057319 18774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d31a02735bfd463c934dcef271464064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31a02735bfd463c934dcef271464064" member_type: VOTER last_known_addr { host: "127.27.46.254" port: 38005 } } peers { permanent_uuid: "8bc45c5cd0b8498ab8d6b55608ca2dcc" member_type: VOTER last_known_addr { host: "127.27.46.253" port: 43805 } } peers { permanent_uuid: "3e0a6e0d484a4eddb53486119b29dcf0" member_type: VOTER last_known_addr { host: "127.27.46.252" port: 38489 } } }
I20260925 04:38:04.057440 18774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:38:04.062232 18785 catalog_manager.cc:1347] Generated new cluster ID: 53b345e56a94424793e78f6a26bcd7de
I20260925 04:38:04.062348 18785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:38:04.085650 18785 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:38:04.087405 18785 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:38:04.102233 18785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d31a02735bfd463c934dcef271464064: Generated new TSK 0
I20260925 04:38:04.103199 18785 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:38:04.141162 18585 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.138717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.27.46.254:38005)
0925 04:38:04.138837 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.138842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.140041 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 04:38:04.140256 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.140261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.140314 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.140369 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.141026 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47324: BlockingRecv error: recv got EOF from 127.0.0.1:47324 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 04:38:04.142624 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.139478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.27.46.253:43805)
0925 04:38:04.139599 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.139603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.141670 (+ 2067us) server_negotiation.cc:407] Connection header received
0925 04:38:04.141840 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.141843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.141891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.141938 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.142505 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55284: BlockingRecv error: recv got EOF from 127.0.0.1:55284 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 04:38:04.144553 18746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.143244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.27.46.252:38489)
0925 04:38:04.143394 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.143398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.143676 (+ 278us) server_negotiation.cc:407] Connection header received
0925 04:38:04.143838 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.143841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.143887 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.143936 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.144428 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59364: BlockingRecv error: recv got EOF from 127.0.0.1:59364 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 04:38:04.148582 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.147168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.27.46.253:43805)
0925 04:38:04.147278 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.147282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.147702 (+ 420us) server_negotiation.cc:407] Connection header received
0925 04:38:04.147891 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.147894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.147962 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.148008 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.148483 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55290: BlockingRecv error: recv got EOF from 127.0.0.1:55290 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:38:04.153283 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.151949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.27.46.253:43805)
0925 04:38:04.152039 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.152043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.152498 (+ 455us) server_negotiation.cc:407] Connection header received
0925 04:38:04.152664 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.152667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.152713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.152760 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.153196 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55306: BlockingRecv error: recv got EOF from 127.0.0.1:55306 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 04:38:04.166817 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.162008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.27.46.253:43805)
0925 04:38:04.162113 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.162118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.165824 (+ 3706us) server_negotiation.cc:407] Connection header received
0925 04:38:04.166043 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.166046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.166101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.166153 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.166705 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55320: BlockingRecv error: recv got EOF from 127.0.0.1:55320 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 04:38:04.174474 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.172965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.27.46.253:43805)
0925 04:38:04.173090 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.173094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.173636 (+ 542us) server_negotiation.cc:407] Connection header received
0925 04:38:04.173813 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.173816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.173863 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.173911 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.174377 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55334: BlockingRecv error: recv got EOF from 127.0.0.1:55334 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 04:38:04.186532 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.184848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.27.46.253:43805)
0925 04:38:04.184977 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.184983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.185590 (+ 607us) server_negotiation.cc:407] Connection header received
0925 04:38:04.185795 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.185799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.185850 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.185906 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.186420 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55338: BlockingRecv error: recv got EOF from 127.0.0.1:55338 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 04:38:04.207309 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.205404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.27.46.253:43805)
0925 04:38:04.205597 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.205602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.206220 (+ 618us) server_negotiation.cc:407] Connection header received
0925 04:38:04.206438 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.206441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.206503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.206572 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.207180 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55348: BlockingRecv error: recv got EOF from 127.0.0.1:55348 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:38:04.243602 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.242083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.27.46.253:43805)
0925 04:38:04.242342 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.242346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.242604 (+ 258us) server_negotiation.cc:407] Connection header received
0925 04:38:04.242799 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.242803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.242867 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.242967 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.243448 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55360: BlockingRecv error: recv got EOF from 127.0.0.1:55360 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
W20260925 04:38:04.311024 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.309773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.27.46.253:43805)
0925 04:38:04.309910 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.309914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.310073 (+ 159us) server_negotiation.cc:407] Connection header received
0925 04:38:04.310194 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.310196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.310268 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.310350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.310881 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55372: BlockingRecv error: recv got EOF from 127.0.0.1:55372 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 04:38:04.442590 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.441371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.27.46.253:43805)
0925 04:38:04.441611 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.441616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.441710 (+ 94us) server_negotiation.cc:407] Connection header received
0925 04:38:04.441839 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.441841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.441891 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.441971 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.442458 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55380: BlockingRecv error: recv got EOF from 127.0.0.1:55380 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 04:38:04.701869 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:38:04.700864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.27.46.253:43805)
0925 04:38:04.701009 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:04.701013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:04.701091 (+ 78us) server_negotiation.cc:407] Connection header received
0925 04:38:04.701195 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:04.701197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:04.701247 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:04.701327 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:38:04.701772 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55394: BlockingRecv error: recv got EOF from 127.0.0.1:55394 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:38:05.010039 18762 catalog_manager.cc:1259] Loaded cluster ID: 53b345e56a94424793e78f6a26bcd7de
I20260925 04:38:05.010149 18762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: loading cluster ID for follower catalog manager: success
I20260925 04:38:05.011950 18762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: acquiring CA information for follower catalog manager: success
I20260925 04:38:05.013113 18762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e0a6e0d484a4eddb53486119b29dcf0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:38:05.028689 18798 catalog_manager.cc:1259] Loaded cluster ID: 53b345e56a94424793e78f6a26bcd7de
I20260925 04:38:05.028759 18798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: loading cluster ID for follower catalog manager: success
I20260925 04:38:05.030267 18798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: acquiring CA information for follower catalog manager: success
I20260925 04:38:05.031338 18798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bc45c5cd0b8498ab8d6b55608ca2dcc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:38:05.220865 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:05.216203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.27.46.253:43805)
0925 04:38:05.216667 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:05.216672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:05.216689 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:38:05.216756 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:05.216758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:05.216878 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:05.217007 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:05.217308 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.217964 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.219244 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.219481 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.219607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:05.219970 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:05.220420 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:05.220561 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:05.220640 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 04:38:05.220700 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":70,"threads_started":1}
I20260925 04:38:05.228576 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:05.224501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.27.46.252:38489)
0925 04:38:05.224771 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:05.224778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:05.224797 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:38:05.225024 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:05.225028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:05.225145 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:05.225289 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:05.225586 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.226087 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.227022 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.227259 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.227386 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:05.227774 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:05.228151 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:05.228288 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:05.228351 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 04:38:05.228406 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260925 04:38:05.235836 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:05.231766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.27.46.253:43805)
0925 04:38:05.232191 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:05.232196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:05.232212 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:38:05.232262 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:05.232265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:05.232316 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:05.232394 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:05.232825 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.233316 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.234399 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.234754 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.234871 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:05.235201 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:05.235279 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:05.235372 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:05.235631 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:38:05.235696 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260925 04:38:05.236239 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:05.231823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.27.46.252:38489)
0925 04:38:05.231931 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:05.231935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:05.232355 (+ 420us) server_negotiation.cc:407] Connection header received
0925 04:38:05.232481 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:05.232483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:05.232561 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:05.232647 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:05.232955 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.233715 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.235030 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.235366 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.235458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:05.235775 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:05.235853 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:05.235923 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:05.236075 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 04:38:05.236128 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:38:05.236969 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:38:05.231906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.27.46.254:38005)
0925 04:38:05.232191 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 04:38:05.232196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:38:05.232212 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:38:05.232296 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:38:05.232299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:38:05.232442 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:38:05.232856 (+ 414us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:38:05.232979 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.233785 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.234864 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:38:05.235202 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:38:05.235649 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:38:05.235801 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:38:05.236516 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:38:05.236677 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:38:05.236759 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 04:38:05.236817 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":136,"threads_started":1}
I20260925 04:38:05.240208 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18512
I20260925 04:38:05.247649 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18589
I20260925 04:38:05.253885 27835 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjz8mnA/build/debug/bin/kudu with pid 18669
2026-09-25T04:38:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3024 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6054 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283169 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST